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ABSTRACT 

This document contains descriptions of the methods 
used by three government agencies (National Institute of Mental 
Health, the 0,S, Department of Agriculture, and the Office of 
Economic Opportunity) in managing practice-oriented research. Each 
agency is treated separately, with a presentation of the steps that 
its managers and researchers actually take in managing research 
activities. The descriptions were obtained by interviewing managers 
and other staff personnel in these agencies and interpreting their 
responses. The document is divided into an introduction and three 
sections for the three agencies. In the Department of Agriculture 
section, the main discussion centers on Cooperative Research Service 
paradigms. The National Institute of Health section focuses on the 
activities of applied research, coordinating centers, funded centers, 
and research and development. The shorter section on the Office of 
Economic Opportunity consists of an overview, summary, and 
description of activities, (JA) 
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PREFACE 



This Working Note contains descriptions of the methods that the National 
Institutes of Mental Health, the U.S. Department of Agriculture, and the Office 
of Economic Opportunity use in managing practice-oriented research. These 
descriptions were obtained by interviewing managers and other staff personnel 
in these agencies -and interpreting their responses. None of the descriptions 
have yet been returned to the agencies for their comment or approval. This 
will be done in the near future. 

This work was done as part of the effort to plan the National Institute 
of Education (NIE) . If authorized by the Congress, the NIE would conduct re- 
search and development in the field of education. This report is one of a 
series on the Institute. The others are: 

o National Institute of Education: Preliminary Plan for the 
Proposed Institute (R-657-HEW) 

o National Institute of Education: Methods for Managing 
Fundamental Research (WN-7676) 

o National Institute of Education: Methods for Managing 
Programmatic Research and Development (1^-7678) 

o National Institute of Education: Organizational and 
Managerial Alternatives (WN-7679) 

o National Institute of Education: Evaluation of Methods 
for Managing Research and Development (WN-7680) - . 
This report only describes the methods that the selected agencies use in 
managing practice-oriented R&D; it does not evaluate their relative merits. 
A comparative evaluation of these methods appears in WN-7680. 

Many of the R&D managers interviewed during this study expressed the need 
for additional study of the methods used in managing non-military R&D in the 
Federal government. The literature on this subject is. slight in comparison 
with the literature concerning the management of industrial and military R&D. 
The principal purpose of these reports, however, is to enable the planners of 
the National Institute of Education to benefit from the experience of other 
federal R&D agencies in developing the NIE's R&D management procedures. 
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I. INTRODUCTION 

PU .RPOSE OF THIS REPORT 

This report describes eight methods that three federal agencies use 
in managing practice-oriented R&D. 

The format of description will be to treat one agency at a time by 
(1) presenting the steps that its managers and researchers actually take 
in managing research activity, and then (2) repeating this presentation 
for the other agencies. The intention is to present the data on which the 
interpretive and evaluative statements made in WN-7679 and WN-7680 are 
based. This approach was taken because agreement on what people do is 
easier to get than agreement on the effects of their actions, or what they 
should do. Agreement on what R&D managers should do is difficult because 
it is ultimately a question of value. 

The description for each agency will be called a paradigm, since not 
every detail and variant in what an agency does will be described. Each 
description is ireant only as a model which depicts the essential steps in 
an agency's R&D management process. 

A step is deemed essential if changing it would significantly alter 
an estimate of the basic philosophy underlying the R&D management process 
being described. By looking at the essential steps, it is easier to infer 
what the basic underlying philosophy is, and how to project it onto a new 
situation, like education R&D. This is, in fact, a meaningful definition 
of a paradigm; that is, the projection of something which is difficult to 
describe onto reality, where its consequences are obser'/able and hence 
describ able. 

The paradigms which will be treated in this report are listed in Table 
alongside the agencies using them. 
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Table 1 



AGENCIES TREATED AND TllE METHODS USED 



TO MANAGE PRACTICE- ORIENTED R&D 



Agency 



Paradigm 



U. S. Departmeiit of Agriculture 



Experiment Stations 
Regional Research Projects 
Special Research Projects 



National Institute of Mental Health 



Applied Research 
Coordinating Center 
Funded Center 
Services R&D 



Office of Economic Opportunity 



Research and Evaluation 



M ETHOD OF RESEARCH 

The data used to construct the management paradigms were obtained by 
interviewing federal R&D managers. Roughly half of the people interviewed 
were program directors and the other half were supervisory and staff 
personnel. Exhaustive coverage of every manager in an agency was not 
atterapted, but rather key personnel and those recommended by key personnel 
were approached for interview. Altogether 24 managers were interviewed, 
some on repeated occasions. In .addition, some data from the academic 
literature and from agency documents were used. A list of the people inter-- 
viewed will appear in the final version of this report. 

The paradigms are a distillation of replies made by managers commenting 
on the nature and importance of their activities. Necessarily, this approach 
to lesearch is vulnerable to biases and sometimes produces information that 
is difficult to verify. Nevertheless, by asking all managers similar 
questions, and by filtering the responses as ' objectively as experience made 
possible, a fair representation of reality is thought to be presented. This 
approach is within the tradition of naturalistic observation as a method of 
research. 

To gain clarity of exposition, some of the auxiliary mechanisms used 
by some agencies to overcome shortcomings in their management processes were 
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omitted. Thus, matching the paradigm descriptions, the agencies inter- 
viewed, and the paradigm evaluations in WN-7680 to conclude that one 
agency does a better job of managing research than another agency is not 
justified, 

TYPES/ OF R&D ACTIVITY 

Practice-oriented R&D includes three sub-types of activity: research, 
development, and evaluation. Practice-oriented research is directed tov^'ard 
understanding the relationship of selected factors to phenomenon Important 
to practice. Typical projects would be: determining the factors that 
effect teacher mobility, determining the factors that effect the rate of 
progress in programmed instruction, or proving relationships between leader- 
ship style and innovativeness in schools. In practice-oriented R&D, develop- 
ment is directed toward inventing a new product, system, or procedure that 
satisfies a need perceived by practitioners. Evaluation , in practice-oriented 
R&D, is directed toward understanding the extent to which, and why, specific 
R&D interventions into practice are working. Evaluations provide information 
that allows more informal choice among alternative courses of action. 

Practice-oriented R&D is distinguished from fundamental research princi- 
pally in the kinds of problems solved. In practice-oriented R&D, the problems 
solved are those faced by practitioners such as teachers, administrators, 
farmers, and medical doctors. In education, for example, the range of topics 
ir»,cludes improving the methods of instruction, developing better curriculum,, 
improving the management of schools, creating new forms of schooling, and 
improving the training of personnel. In fundamental research, the problems 
solved are less directly derived from practice and more concerned with under- 
lying natural phenomena. In education, examples of fundamental research 
studies would be measuring the perceptual capacities of infants, developing 
theories of language acquisition, and studying genetic effects on behavior. 

In comparison to programmatic R&D, practice-oriented R&D is similar in 
some ways and different in others. They are similar in that both are con- 
cerned with solving practitioners' problems. They are also similar in that 
both include research, development, and evaluation as constituent activities. 
They are different in that programmatic R&D involves a larger scale, more 
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planned attack on a single problem. In most cases, programmatic R&D is 
used when the solution to a problem is urgently needed. Practice-oriented 
R&D is more directed toward advancing the state-of-the-art in problem areas 
of continuing importance to practice. 

TYPES OF MANAGEMENT ACTIVITY 

To facilitate presentation, the practice-oriented R&D management 
process will be broken into three types of activity: 

o Program Planning, 

o Program Development, 

o Program Evaluation. 
These categories are deliberately chosen to group together qualitatively 
similar management activities. 

Program Planning management activity is defined to include all the 
actions taken to foster, detect, and incubate new and ill- formed programs. 
Also included are the procedures for deciding which new programs will be 
added to the set of ongoing streams of activity. One example of such a new 
area is OEO's effort to understand whether vouchers are a desirable means of 
financing and providing educational services. While this idea is based on 
concepts from pure economic theory, the whole range of effects which might 
occur from implementing the concept in practice have never been studied. 
In fact, the alternative means for implementing the concept had only been 
crude-ly considered prior to OEO's interest. Thus, educational vouchers are 
a wholly new idea for practice, not clearly superior to existing methods, 
but potentially of very great practical utility. These characteristics are 
typical of new program areas in practice-oriented R&D. 

Progra^n Development is defined to be the activity of managing the 
continuous process of refining and elaborating knowledge and practical 
capability in a program area. As a management process, program development 
is typically, though not always, an iterative and continuing sequence of 
stages involving: 

o assessment of needs, 

o generation of project ideas, 
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o selection of projects to support, 
o monitoring of project performance, 
o evaluation of project outcomes, and 
o utilization of results. 

In various management paradigms, these stages are managed in different, 
ways. Sometimes they are done by program directors, sometimes by panels of 
scientists, and sometimes not at all. In most of these stages action pro- 
ceeds concurrently in several of the stages. This activity is called Program 
Development because from program management's perspective, practice-oriented 
R&D is evolutionary and expositional in nature. The goal is adding fine 
structure and precision to a basic idea, and this happens through a sequence 
of project generation, project selection, and outcome events that unfold 
over a period of time. 

Program Evaluation is the management activity of assessing what performers 
have accomplished at some point in time and the Judgment of what ought to be 
done next. Ways in which agencies accomplish this management activity is the 
third topic which will be discussed in this paper. 

Discussion of management methods will focus on what is done at the 
program director's level, and only occasionally at the upper levels. This is 
necessary because of the decision to describe procedures that managers use in 
practice. Interactions at the higher levels are more political, and thus sub- 
ject to greater variation and personality dependencies. Not much insight into 
ways of allocating a budget between R&D on teacher education and school 
administration is gained by looking at the procedures agencies use. The 
mysteries surrounding how these kinds of decisions are made is treated in the 
science policy and political science literature. Attention here will center 
on what goes on at the interface between the performer and his immediate 
manager. In some agencies this latter individual is called a program officer, 
or science administrator. 



-6- 



II, U.S. DEPARTMENT OF AGRICULTURE 

OVERVIEW 

The U.S- Department of Agriculture (USDA) has managed a large-scale 
R&D system for a longer time than any other federal agency. USDA began 
supporting research at its inception in 1862. At the present time there 
are six research services in three agencies supporting agricultural R&D. 
A list of these services appears in Table 2. 



Table 2 

RI5SEARCH SERVICES IN THE U.S. DEPARTMENT OF AGRICULTURE 



Agency 


Service 


Science and Education 


Agriculture Research Service 
Cooperative State Research 
Service 


Agricultural Economics 


Economic Research Service 
Statistical Reporting Service 


Rural Development and 
Conservation 


Forest Service Research 
Farmer's Cooperative Service 



Only the Cooperative State Research Service (CSRS) will be described in 
detail in this report. 

In addition to the agencies listed, there are two other USDA agencies 
that do not support R&D: International Affairs and Commodity Programs, 
and Marketing and Consumer Services. 

Agriculture's R&D system has three major segments. One is intra- 
mural research, which was started in USDA's first year. Since then it 
has been used as a means for improving agricultural practice and pro- 
viding a scientific basis for establishing agricultural regulations. 
Most of the intramural research in USDA is managed by the Agriculture 
Research Service (ARS) . Currently, ARS includes more than 4,500 scientists 
in 150 laboratory locations across the country, requiring an annual ex- 
penditure of over $170 million. 

The second major segment: of Agriculture's present R&D system was 
added by the Hatch Act of 1887. This act provided for S tate Agricultural 
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Experiment Stations (SAES) at the land grant colleges. These institutions 
were to be a means of focusing research on the problems of a region and 
coupling this research with the education function of the land grant 
colleges. The experiment stations are funded by USDA^s Cooperative State 
Research Service and by other federal and state agencies. Currently, 
there are 53 SAESs receiving $56 million from CSRS, $29.4 million from 
other federal agencies, and $138 million from state agencies. In addition 
industry contributes $11 million. 

The SAES segment of Agriculture's R&D system contains three sub- 
segments, which have been added at separate times over the years. The 
largest subsegment (currently about 95 percent of the state and federal 
funds spent by stations) comprises projects undertaken by researchers at 
the station because of their o\m scientific interest or because- a problem 
exists in the state that the station serves. These projects are supported 
by money alio ted to the states on a formula/matching basis by the federal 
government or from state appropriations. By design, there is very little 
federal or regional (multistate) control that can be exercised over the 
allocation of this money. These projects are the oldest experiment station 
activity, descending in concept from the first two .experiment stations es- 
tablished in Connecticut in 1875 and North Carolina in 1877. The original 
intention of the North Carolina station was to answer farmers' questions 
about commercial fertilizers by using the "latest achievements in science." 
This purpose has been repeated and continued in principle in every sub- 
sequent station. This subsegment of activity will be called Experiment 
Stations research. 

The second largest subsegment of the SAES program is Regional 
Research Projects , which was added in 1955. The Station Directors and 
Congress came to realize that common problems existed among the states, 
but too often each state was working independently, in solving them. 
Regional Research Projects, in which a group of scientists from the 



U.S. Department of Agriculture, After A Hundred Years: The Year- 
book of Agriculture^ 1962, G.P.O., 1962, p. 26. 
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several states plans a research project together, was instituted to 
correct this deficiency. At first the scientists controlled these 
projects, but in recent years the Directors have begun to use them as 
a principal lever for setting research priorities in the entire system 
of the stations, /ibout five percent of the budget in SAESs is allocated 
to Regional Research Projects. 

A third subsegment of the experiment station activity, called 
Special Research Projects , has been added more recently. The purpose of 
Special Research Projects is to mobilize available talent quickly in 
response to a critical problem that must be solved in a short time. Both 
the formula-supported projects and the Regional Research Projects have 
turned out to be inadequate to this task. A recent example of such an 
effort is the crash program to solve the com blight problem. Less than 
five percent of the federal support to SAESs is spent on Special Research 
Projects . 

At the federal level these three subsegments are managed by the 
Cooperative Sta Ci fi Research Service , an agency in the Science and Education 
directorate of the U. S. Department of Agriculture. Because it facilitates 
understanding they will be described as three distinct programs within CSRS. 
But J there is danger that each will be be seen as a separate activity. CSRS 
stresses the opposite case; that the three are highly interrelated.. It 
is a tenet of management in CSRS that none will function well without 
the other in place. 

The third major segment of Agriculture's R&D system is the Extension 
Service , which was authorized by the Smith-Lever Act of 1914. In the early 
1900s the SAESs had begun to recognize that simply doing good research, 
even practice-oriented research, was not enough to guarantee its implemen- 
tation in practice. A large effort to publish and distribute R&D results 
in written form had not worked, nor had attempts at sending out researchers 
to lecture farmers. The method that has been used by the Extension Service 
has proven to be much more successful in increasing the rate at which research 
results are adopted than the old methods. The Extension Service method is to pro- 
vide, at least one agent for every county or county-sized area of the county who 

ERIC 
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is responsible for solving problems brought to him by people in his area, 
and also showing people that they have problems , At present there 
are 11,000 of these county agents throughout the country. When one of 
these county agents needs information about how to solve an agricultural 
or community problem in his area, he contacts the SAES to which he is 
administratively linked. The agent's points of contact are the exten- 
sion specialists , who work in each SAES . On the average there are 80 
extension specialists per SAES. Each specialist is responsible for 
knowing all the significant research results in a limited problem area 
that have been produced at his own station or at any other station or 
agricultural research laboratory. The specialists respond to agents' 
requests in a variety of ways from sending literature to making a personal 
visit. For difficult problems, the specialist may arrange to have station 
researchers sent to the field, or have a. project to solve the problem 
begun in the SAES. USDA believes that county agents must have this research 
base available if they are to be successful in improving practices in the 
field. 

The extension function is funded by USDA through its Extension 
Service, which is part of the Science and Education agency, and by state 
and local government, which must match the federal contribution. There 
is also a counterpart of the federal Extension Service called a State 
Extension Service associated with each SAES. These State Extension Services 
administer the county agent program, and coordinate program thrusts at the 
county level that are initiated at the federal level by the Extension 
Service. 

\ftile the county agent's roles are crucial in the Agriculture 
R&D system, they w:-**^.! ':ot be discussed further in this report except insofar 
as they interface will*. SAES research programs. This restriction is necessary 
because this report is limited to systems for managing R&D. 

CSRS's system of SAESs is unique among federal agencies in several 
respects, 

" Fi rs t , the system includes a fii'L'L TonQQ oj^ iTi'tQVQOTiTi&Q'b&d J^&D 

activities. The range covers Research, Development, and Extension. 
CSRS believes that without Research and Development, Extension 
withers in effectiveness; and without Extension, R&D strays from 
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solving "real" problems. Performers in each of these activities 
are administratively linked together or collocated according to 
patterns worked out over a period of time. These patterns vary 
over the country, but they are much more uniform and established 
than in any other R&D system. 

CSRS funding is institutional in nature, and is not to any 
large extent awarded by project. About half of the R&D money 
goes to stations by formula and most of the rest is spent intra- 
murally . 

^ The agricultural experiment stations are funded jointly by the 
states, the federal government, and private sources. Whereas 
only one-to-one matching is required, non-CSRS sources actually 
supply four times the federal contribution. 

^ In the experiment stations, almost all personnel perform two or 
three roles in the R&D system. The primary roles performed are: 
research in the station, teaching in the university, and providing 
extension services for the field. 

^ The basic work unit is a multidis dp Unary j problem-oriented teani 
composed of both basic and applied scientists working on a specir 
fic agricultural or community problem. 

^ Unlike other federal agencies, CSRS funds no scholarships or training 
grants. Students work as assistants on research projects, but 
there are no direct instruments employed for research training. 

SUMMARIES OF CQQPEI?ATIVE RESEARCH SERVICE PARADIGMS 

EXPERIMENT STATIONS 

General Characteristics 

Primary output: Additions to knowledge about 

agricultural problems, and 
products or techniques that 
advance agricultural practice. 
Much of this R&D is directed to 
local problems within a state. 

O 
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Mechanism of support: 



Managerial emphasis; 



Staffing plan: 



The station director receives 
population-based formula funds 
from the federal government 
and regular state appropriations, 
which he allocates to projects 
in the station. 

At the federal level control of 
program development is exercised 
by conducting program evaluations. 
Program and project planning is 
done by scientists at the working 
level in cooperation with peers, 
supervisors, and extension per- 
sonnel. 

Most station researchers have split 
assignments; conducting research 
and teaching in the university • This 
is possible because all stations are 
collocated with a land grant univer- 
sity. At the federal level, program 
managers serve full time. Most 
have a background in experiment 
station research. 



Program Planning 

Sources of new program ideas; 



Mechanism for planning: 
Coordination: 



Each station is active in most 
program areas, so new programs 
rarely is tart. Most new programs 
result from user group pressures. 

No formal mechanism employed. 

Peer group communications and 
Regional Research Projects Program 
are principal means of coordination. 
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ProRrain Developinent 

Sources of project ideas: 

Means of proposal review: 



Allocation of budget: 

Monitoring of performance: 
Evaluation of outcomes : 

Program Evaluation 

Mechanism of evaluation: 

Timing of evaluation: 

Implementations or results: 



ERLC 



CSRS has little control over 
program priorities. 

Scientists develop their own 
project ideas in consultation with 
peers, program leader, department 
chairman, and station director. 

Scientists prepare a Project 
Outline for each project they 
undertake. The Project Outline is 
a charter agreed to by department 
chairman, station director, and 
CSRS program director. 

Little direct federal influence. 
Station director is final authority. 

Not done formally. 

Project Outlines have finite 
lifetime, after which review 
process is repeated. 

Two-day site visit by CSRS program 
direc'uor and scientific peers. 

Each program is evaluated a minimum 
of once every four years . 

Review is conducted in interactive 
style between site visit team and 
the station's researchers. Intent 
is that researchers will meld re- 
viewers ' comments with their own 
judgments in planning projects. 
Station directors see the review 
results , too. 
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REGIONAL RESEARCH PROJECTS 

General Characteristics 

Primary output: Additions to knowledge about 

multistate agricultural problems 
and products or techniques that 
advance agricultural practice* 

Mechanism of support: The station director receives 

formula funds from the federal 
government to spend on projects 
in which several states perform 
cooperatively , 

Managerial emphasis: Regional projects are" a means by 

which the Regional Associations of 
Directors and CSRS influence the 
directions of agricultural research 
in the state agricultural experi- 
ment stations. 

Staffing plan: Regional projects are performed by 

experiment station researchers 
working in their own labs , and 
managed by a committee of repre- 
sentatives from each of the par- 
ticipating stations • Intra-project 
priorities are set by the station 
directors. 

Program Planning 

Sources of new program ideas: The Regional Associations of 

Directors select top priority 
problem areas for development into 
regional projects. 



V 



Mechanism for planning: 



Coordination : 



Task forces of scientific leaders 
are convened to sketch out desirable 
projects in each program area. 

Principal responsibility of the 
CSRS program director serving 
on a task force is to bring 
knowledge of all current activity 
relevant to the planning sessions. 
CSRS has no control, however, over 
program priorities. 



Program Development 

Sources of project ideas; 



Means of proposal review: 



Allocation of budget: 



Within a regional project, which 
is almost large enough to qualify 
as a program, individual sub- 
projects are generated by a Tech- 
nical Committee of representatives 
from each participating station. 

The Project Outlines prepared by 
the Technical Committee are re- 
viewed first by the Regional 
Research Committee and the Re- 
gional Association of Directors, 
and then at the federal level by 
the Committee of Nine, a national 
level committee of Directors. CSRS 
assists the Committee of Nine in 
its review responsibilities. 

During the preparation of the 
Project Outline each station 
director decides how much of his 
formula-fixed budget of regional 
funds to spend on each project. 



-15" 



Monitoring of performance; 



When added up, all his regional 
commitments must match the 
supply of funds available from 
CSRS. 

The Technical Committee and Ad- 
ministrative Advisor, an appoin- 
tee of the Regional Association 
of Directors, are responsible 
for monitoring performance. The 
Committee of Nine reviews each 
project one or two years after 
its initiation. 



Program Evaluation 

Mechanism of evaluation: 



No means of evaluation is employed. 



SPECIAL RESEARCH PROJECTS 



General Characteristics 



Primary output: 



Mechanism of support: 



Managerial emphasis; 



Staffing plan: 



Rapidly achieved solutions to 
critical national or regional 
problems. 

Finite duration grant to project 
team. 

An intensive effort is made to 
determine research priorities 
before awarding grants. This is 
done cooperatively by CSRS and 
potential participants. 

Most of the performers are station 
researchers who defer other pro- 
jects to work part- or full-time 
on the special project. 
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Program Planning 

Sources of new program ideas ; 



Mechanism for planning: 



Coordination : 

Program Development 

Sources of project Ideas: 

Means of proposal review: 



Allocation of budget: 



Monitoring of perf oviTiance ; 



No formal mechanism for detecting 
problems. Reliance placed on 
usual Internal and external 
communications. 

Program director assigned full- 
time to problem. He runs a con- 
ference of potential performers 
to explore priorities for spending 
research money on the problem, 
and Investigates the question 
himself • 

Program director links to similar 
efforts by companion agencies. 



Potential performers submit 
their ideas when requested by 
the program director. 

Program director and agency staff 
evaluate prospective performers 
based on quality of ideas contri- 
buted in planning stages and com- 
petence to solve their assigned 
problem quickly. 

Ultimately decided by program 
director; but conference of the 
station directors which will most 
likely be doing the work is run 
to assist in setting priorities 
among sub-tasks. 

Program director keeps in weekly 
contact. 



Evaluation of outcomes : 



Not done in formal way. 



Program Evaluation 

Mechanism of evaluation: Not done in formal way. 

EXPERIMENT STATIONS PARADIGM 

Program Planning 

The total research activity of the state agricultural experiment 
stations has been divided into 98 problem-oriented research categories 
by the USDA. Each of these categories is called a program, but since 
work is largely planned at the scientist level, programs are more a 
managerial artifice than coherent, coordinated sets of projects. Typical 
examples of these "programs" as they are defined at the federal level are: 

° Management of salinity and saline soils 

° Appraisal of forest and range resources 

° Control of insect pests of field crops 

° Improvement of biological efficiency of fruit and vegetable 
crops 

° New and improved meat and dairy products 

° Causes and remedies of poverty among rural people. 

Each program is an aggregate of all the projects conducted in all 
the state stations that fit the program's definition for inclusion. Any 
one station typically has a few projects in almost every program; but 
there is no organized attempt to coordinate projects in the same program 
in different stations with each other. Each station plans its own version 
of the programs on which it is active, often combining several of them 
into one. As a consequence, a station's programs do not necessarily 
correspond one-to-one to the categories set down by USDA. A station's 
research activity might better be described as a collection of over- 
lapping program areas. 

The organization of a state station more closely resembles a collec- 
tion of overlapping groups than a rigidly compartmentalized and hierarchi- 
cal arrangement. Projects consist of one or more of the station's research 
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working on a part-time basis. A mixture of disciplines may be involved 
in a project. The rest of the time these researchers are working on- 
other projects, teaching in the university, or serving as extension 
specialists . 

Since each station has some projects in most of USDA's 98 program 
areas, new programs are rarely started. For this among other reasons, 
Program Planning, as the term is user' here, is not a frequent activity. 

But when he needs to, the statioii director has substantial, but 
certainly not unlimited, authority to start new programs. 

His authority is checked mostly by budget restrictions. The 
station's budget is an aggregate of funds from a variety of sources, 
principally the federal government, state government, and industry. Trie 
station's federal funds are determined by a formula based on rural popu- 
lation, so the director is virtually free of federal control in allocating 
this money. The remainder of the station's money comes as appropriations 
from the state government, or as grants or contracts from industry. Support 
from either of these sources can be increased if an attractive new. program 
is offered. 

Program Development 

Most of the research activity in a state experiment station falls 
in the category of Program Development. 

The process receives its basic direction from the ideas developed 
by scientists at the working level rather than from the station director 
or other administrators. However, the scientists are influenced in their 
generation of ideas by many sources. 

The first source is the extension specialists, who are collocated 
with researchers and every day ask them questions derived from practi- 
tioners' problems. These questions influence the researcher's choice 
of problems to work on. 

Another source is the station director, who may urge the researcher 
to a different research approach because of problenis the director has 
encountered in his travels and discussions. For example, the objective 
that researchers pursued in pesticide research for years was to ma>:imize 
persistence and lethality. Now, the directors are changing this to 
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shorten persistence, and minimize the amount of substance applied. 

A third source, and often the most frequent one, is research 
program leaders. Within a station there are a number or senior re- 
searchers who function as "program leaders." Generally there is one 
program leader per ten researchers in each station. They are of two 
principal kinds: (1) those who are semi~of f icially designated as such 
(it is written into their job description) , and (2) those who have 
mastered a broad field of agricultural science and act as gatekeepers. 
These individuals assume responsibility for coordinating and shaping 
the research program in their area. 

As the first steps in gaining approval for a project, a scientist 
(or scientists) writes up his (their) idea in a format called a Project 
Outline . 

CSRS requires that, in no more than a few pages, a Project Outline 
should: 

(1) Justify the importance of the selected problem to the agri- 
culture of the state, and science, 

(2) Summarize previous, pertinent work on the problem, 

(3) State research objectives in logical order, 

(4) Indicate experimental methods that will be used, and 

(5) Estimate resource requirements. 

The Project Outline is a charter between the station and the proposing 
researcher to conduct research in a delimited area for a finite nuniber of 
years (usually less than five). At the end of this period, or sooner if 
changes are needed, the Project Outline is rewritten. In writing the 
Project Outline, scientists consult with appropriate program leaders and 
the director's office, and work out a compromise that balances the interests 
of the station with the interests of the researcher. 

Once the station director, the appropriate program leaders, the 
researcher's department head, and the researcher agree on a Project 
Outline, it is sent to CSRS in Washington, D. C, for review and approval. 
CSRS reviews all Project Outlines in the same way whether the project is 
federally or state funded. The appropriate CSRS program director reads 
the Project Outline, checking mostly to see if the proposed work is mundane. 
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obsolete, ill-conceive.d , or amounts to research on methodology not cl'^arly 
related to an agricultural problem. Fine lines over what is and what is 
not acceptable have developed over the years. For example, with respect 
to the last criterion, research on soil testing methodology is appropriate, 
but research on the foundations of probability is not. The program direc- 
tor vetoes Project Outlines he thinks are too weak to merit support. 

The program director does not veto very many Project Outlines, just 
enough to keep the system honest. Typically, five percent or fewer are 
vetoed. From experience, the program director knows that if he vetoes 
many proposals, the station is likely to fund most of the vetoed projects 
from its other sources to emphasize its autonomy. There is no legal authority 
granting veto power to the program director, simply the statement that the 
"Secretary is charged with proper administration of this Act..." The 
understanding that has been reached between CSRS and the stations is 
representative of the mode of operation of the agriculture R&B system. 
Management is viewed as a cooperative enterprise in which all parties 
should have a voice. 

Vetoed proposals are reworked by the proposing researcher, or research 
team, as the case may be, and resubmitted. More than 90 percent of all 
Project Outlines resubmitted are eventually approved. 

Once a project is accepted by CSRS, monitoring and supporting per- 
formance is the responsibility of the station staff, particularly the 
program leader and the department chairman. No forrialized method of re- 
viewing performance has developed over the years. 

Project outcomes are evaluated at the expiration date of the Project 
Outline. A researcher's output, the need for more of the same work, and 
other factors are reviewed informally by the researcher's peers and super- 
visors. A decision is then made to continue the Project Outline or submit 
a modified one. "Newly hired" researchers receive a more severe evaluation 
at this stage than researchers up for renewal, since a decision on tenure 
in the department is made at the same time. Tenured researchers are more 
likely to renew their old Project Outline without significant change. 

A diagram of the program development process appears in Figure 1, 
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Program Evaluation 

Even though it is only peripherally involved in Program Development, 
CSRS influences it by conducting program evaluations. The means of doing 
this is a site visit by CSRS program officers and peer scientists. 

The format of site visits has evolved over the years, and is still 
evolving because CSRS continually experiments with new techniques (they 
are experimenting now with non-paid reviewers) . Fifteen years ago, the 
revie^^r was only financial, since CSRS program directors went over expen- 
diture reports with the station director. It was then realized that 
financial review was a job for auditors, and that most of the problems 
could be solved with a decent project reporting system. The site visits 
were transformed into reviews of progress-to-date. This, too, was soon 
found to be unproductive. Now, the site visits concentrate on reviewing 
the planning process and looking into the future. All the time is spent 
discussing future direction for the programs. 

The site review team consists of one CSRS program director, plus 
one CSRS visiting scientist; and, depending on the comprehensiveness of 
the review, up to four scientists from other stations. Reviews are con- 
ducted at a continuum of levels from individual progr^^ms (activity in one 
of the 98 categories) to comprehensive reviews of an entire station. The 
format is probably converging to a point nearer the comprehensive end of 
the scale. 

CSRS's objective is to include the activity in every program category 
in every station in at least one review every four years. 

Reviews are conducted in an interactive style and last for two or 
three days. The attempt is to review by dialogue rather than to sit in 
judgment, although this varies with the program director's style. Typically, 
the first session is a group meeting and then discussions are held with 
individual scientists. Most of the talk is about specific projects and 
where they are going next, but concern is at the program Travel. For 
example, the purpose of a review might be, "Should the X station still 
be involved in plant breeding?" But talk Would be about ongoing and . 
possible projects. 

In order to avoid devoting time to the history of a station's program, 
the CSRS program director spends two or three months part-time before a 
site visit studying the station's current and past research program. ' 
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does this by extracting the station's whole research program from the 
Current Research Information System, a computerized information system, 
and by reading final reports. Summaries of this review are sent to the 
site visit team. CSRS suggests that stations prepare for the review, too. 
Although a report is submitted to the station director and CSRS, the 
biggest effect of site visits is on what the scieutisuti decide to do in 
the future. Station scientists think about what's said and "meld the 
thougjits" in with their own. The result is reflected in the Program Develop- 
ment process. 

CSRS chose this style of evaluation because its method for managing 
R&D is to recognize that R&D is a process involving many phases and per- 
forming instruments. Sometimes what is done managerially in one phase of 
the process will more strongly affect what happens in another phase than 
operating directly on the latter phase. Thus, evaluating the potential 
of research programs at multiyear intervals is believed to have as much 
effect on program development as would being strongly involved in project 
planning. Moreover, the conduct of regional research projects and special 
research projects will also be affected by such evaluation, even though 
they are not directly under scrutiny because the same scientific staff 
participates in them. 

ORGANIZATION 

Structural Relationships 

The organization chart of a typical experiment station appears in 
Figure 2* 

Staffing Relationships 

CSRS Program Director . With rare exception, all hav^ Ph.D. or equiv- 
alent degrees. Most come originally from the research bench in an experiment 
station. A portion of the assistants are scientists on visiting appointment. 

Dean, School of Agriculture . The dean and his immediate staff are 
.. responsible for the instructional program of the university. 

Director, State Agricultural Experiment Station . The director and his 
immediate staff are responsj.ble for all research activity financed by the 

O 
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t^tation. Very little other agricultural research is conducted by the 
university, although some departments, such as zoology, do related research. 

The direcror is nominated by the station and approved by the 
Secretary of Agriculture. None has ever been disapproved. 

Director, State Extension Service . The director and his immediate 
staff are responsible for managing the network of county agents, and res-* 
ponding to federal extension initiatives. They are also responsible for 
coordinating the work of the extension specialists as it relates to county 
agents' needs. 

Department Chairman . Department chairmen are administratively res- 
ponsible for station scientists and extension specialists in their depart- 
ment. In addition, the department chairmen coordinate the research activity 
in their departments, the extension activity, and the relations between ex- 
tension and research. The department chairman reports to three people: 
the Station Director, the Dean of Agriculture, and the State Extension 
Director, 

Station Scientists . Station scientists are assigned administratively 
to a department, but most have the dual role of teaching in the university 
and doing research. Normally, station scientists work simultaneously on 
two projects of their own and one cooperatively with two to four other 
scientists or extension specialists. 

A job description agreement specifies the scientist's specific 
responsibilities as to division of labor^ and managerial authority, but 
it prohibits pay for consultant services. The agreement is reached before 
a man is hired, and is a basis for performance review by the department 
chairman. 

Extension Specialists . Extension specialists are assigned to a 
department. The extension specialist's responsibility is to keep up with 
what is knox^m in his knowledge specialty, so that he can respond when 
county agents bring in a problem. To keep up-to-date, the extension 
specialist spends part-time doing research or teaching. His division 
of responsibilities is specified in a job description in the same manner 
as the scientists'. He is forbidden to receive extra pay for consulting. 



ERLC 



-26- 



REGIQNAL RESEARCH PROJECTS PARADIGM 
Program Planning 

Regional Research Projects are a principal means of setting prioriti 
in the experiment station's research program. The method is to select a 
problem that is important to a region or part of a region, and then have 
selected scientists from participatinfj stations solve it cooperatively. 
Through the discussion necessarily generated, the selected scientists 
get a chance to set priorities in their own minds, and take into account 
what other stations are doing. Their attention is drawn to regionally 
important problems, and when they talk to their colleagues back at the 
station, this effect also transfers to nonparticipants . When the new 
thinking is reflected in proposals for formula-grant supported research, 
the priority-setting effect of regional research is achieved. 

Until ten years ago, the ideas and impetus for regional projects 
came from the working scientist level, but now the Regional Associations 
of Directors play a primary role. While methods for planning regional 
projects are still being explored, one pattern the Regional Associations 
of Directors have followed is to set aside a portion of their quarterly 
meetings for discussion of important regional issues and to decide on a 
lisL of important regional problem areas. As a starting point, the 
directors sometimes use the 32 task force reports developed during the 
National Program of Research planning effort in 1966. Another source is 
issue papers drawn up by the Regional Association's staff, the Regional 
Research Committee. Each year, the directors choose one or two problem 
areas for further development. 

The Regional Association of Directors then appoints a task force of 
department chairmen and leader scientists for each problem area selected. 
The task force members are selected on the basis of recognized scientific 
excellence and breadth of view. The task force's job is to decompose 
the problem area into a set of research objectives, each one to be a 
candidate for a regional research project. The task force indicates 
priorities among the projects suggested, and estimates the resources 
roost likely to be required for each one. 
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In the meantime, each director compiles his obligations for regional 
research activity during the coming year, and sends it to CSRS. CSRS 
computes the difference between Regional Research Funds already committed 
by all the directors and the amount available from Congress, and dis- 
tributes the surplus to each station in proport-'J '^n to the previous year's 
allocation. Thus, each director knows how much of a commitment he can 
make to nc^w regional research activity. 

The task force's report is reviewed by the Regional Research Comnittee, 
and then both repo^rts are submitted to the Regional Association of 
Directors. The assembled directors weigh the list of projects recommended 
by the task force and the RRC. They select a subset of these projects that 
includes those felt to be the most important ones and that just exhausts 
the supply of available regional funds. An administrative advisor is 
appointed for each of the selected projects and he is authorized to 
call the first Technical Committee meeting. A diagram of the planning 
process appears in Figure 3. 

CSRS influences the planning process by having one of its program 
directors serve as a contributing member of the task force. This program 
director is technically qualified and orings CSRS priorities and percep- 
tions to the planning activity. 

Program Development 

The Program Development process is managed to a large extent by 
a Technical Cominittee composed of representatives from the participating 
stations. Each station director appoints one or more representatives 
to the Committee af^er spending considerable time discussing possible 
members with the administrative advisor of the project. The Technical 
Committee meets for the first time at the call of the administrative 
advisor. 

The administrative advisor introduces the Committee members to 
regional research, describes the project's history and objective, and 
emphasizes that the Committee's concern should be with technical matters, 
not the allocation of monies to stations. At the first meeting the 
Committee discusses specific sub-subproblems of the assigned overall 
problem, ranks these subproblems by importance to agriculture and their 
amsnability to regional research. Ea^ch t opresentative indicates his 
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interests and then a tentative sketch of the regional project is produced. 
Sometime during the first session a chairman and other officers ate 
elected by the Technical Committee members. From then on, Technical 
Committees are run by the elected chain^ian. 

The Committee's next task, done over a period of months, is to 
develop a detailed project plan, called a Project Outline. The Project 
Outline specifies project duration (not more than five years), objectives, 
procedure, current and previous work, and organization. The procedure 
section should indicate initial plans to achieve each project subobjectives 
methods for collecting data, schedules, and means of coordinating activity. 
The allocation of tasks and responsibilities to stations participating in 
the project must be explicitly stated. 

CSRS influences the Program Development process through its delegate 
on the Technical Committee. CSRS makes the representative responsible 
for knowing the state-of-the-art in the problem area of the project, and 
by employing this knowledge to keep the Technical Committee from being 
captured by a strong personality or from straying off the track. 

Each representative must discuss what resources his station will 
commit to the project with his station director before it is included in 
the plan. A good deal of bargaining can occur over this matter, for the 
station director must allocate his fixed regional funds to several regional 
projects optimally, and the researcher usually wants maximum participation. 
The administrative advisor, a delegate of the Regional Association of 
Directors, plays a coordinating role in setting the allocation of tasks 
among the stations. He tries to see that tasks go where competence lies. 

Completed Project Outlinej ^re sent to the Regional Research Committee 
for preliminary review and then to the Regional Association ©f Directors. 
Changes desired by the Directors are incorpc^rated by the Administrative 
Advisor, and then Project Outlines are sent to the Committee of Nine, 
which is the legal authority for paying regional projects. 

Incoming Project Outlines are cissi^ned to one Committee of Rine 
member, and a CSRS program ('''rector for in-depth, written review. In 
writing his review, the Committee of Nina- member can employ any consultants 
that he pleases, even the opinions of CSRS. In the Committee meeting. 
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proposal? are discussed in turn> and then a decision made to approve, dis- 
approve, or conditiocally approve by majority vote. The Committee rarely 
disapproves a project^ but frequently gives only qualified approval. For 
example, the Committee might grant approval for only one year instead of 
five, with the stipulation that particular issues be resolved by the end 
of that year. Or, the Committee might eliminate parts of the Project 
Outline. In total, the Committee declines enough proposed work to keep 
the planning activity earnest. The vigor with which the Committee reviews 
Project Outlines varies greatly with its constitution, a matter over which 
GSRS has no control. 

Upon receiving the Committee of Nine's approval, the station directors 
initiate work on their portions of the regional project. During project 
performance the administrative advisor carries much responsibility for 
keeping the project on schedule and working to th^- Project Outline. He 
confers with the CSRS program director on the project, checks on the 
adequacy of participation, and counsels individual researchers. He is 
responsible for calling at least one meeting of the Technical Committee 
per year or more if needed. 

At these meetings, the Technical Committee reviews progress of par- 
ticipating scientists to see if it confonns with planned objectives and 
procedures. To assist in this review each participant is required to send 
a written report of his results to all the Committeemen before the review 
session. The Committee also discusses the work planned for next year and 
the desirability of changing the work schedule written in the Project 
Outline. If reallocation of a station's resources are required, the 
administrative advisor negotiates the differences with the station 
director. A third item of business is review and approval of the project's 
annual report, which is prepared in advance by the chairman. The approved 
report is sent by the administrative advisor to CSRS, the Regional Research 
Committee, and the Directors. 

The Technical Committee also designates one of its members or the 
chairman as Project Coordinator. On large projects the Conunittee may hire 
a Project Coordinator. The coordinator's job is to maintain contact with 
the cooperators through correspondence and personal visits. He assists 
in the preparation of experimental protocols so that findings will blend 
into a regional analysis, and participates in the assembly and analysis 

ERLC 
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of data to assure uniform procedures* 

The CSRS program director's responsibility in the regional project's 
development phase is to assist the administrative advisor in performing 
his role by maintaining close personal relations and a mutual undcirstanding 
of project evaluations and problems. The CSRS program director and the 
administrative advisor are the chief communicators between the researchers, 
directors, cooperating agencies, and CSRS. 

When the project is one year from termination, the Technical Committee 
must decide whether, during the next five-year period, the project should 
be terminated, extended with the same Project Outline, or revised and 
performed under a modified outline. The administrative advisor is res- 
ponsible for organizing the required meetings and assignments for executing 
this task. If the Project Outline is to be significantly changed, the 
process is not unlike the one followed in preparing a new Project Outline. 
If the changes are minor, shortcuts in this procedure are in order. In 
any case, both requests for extension and revised Project Outlines must 
go through the same evaluation steps as first-time proposals. Requests 
to merely extend the Project Outline are not received by the Committee of 
Nine. 

The administrative advisor is also responsible for seeing that the 
Technical Committee publishes its findings broadly. Station publications , 
journal articles, and briefs are often used mediums. In addition, the 
Technical Committee roust submit a final report at the end of its funding 
period. 

Procedures for evaluating regional projects vary with the region and 
the project, with no clear pattern emerging except that after the first 
one Of two years of a project's life Che Committee of Nine reviews its 
progress. Their review is based on a report from the administrative advisor 
and comments from the participating CSRS program director. 

A description of the program development process appears in Figure 4. 
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ORGANIZATION 

Structural Relationships 

An organization chart for the structure used in managing regional 
projects appe^irs Figure 5. 

Staffing Relationships 

CSRS Program Directors , With rare exception all have the Ph.D. degree 
or equivalent. Most come originally from the research bench in a station, 
A portion of the assistants are scientists on visiting appointment. 

Committee of Nine . Membership is elected by and represents the 
Regional Associations of Directors. Each Regional Association elects 
two rppi^esentatives to serve for three years. The ninth member is an 
administrator of home economics research in an SAES, elected at the 
annual meeting of the National Association of State Universities and 
Land Grant Colleges. The Committee meets tri-annually to discuss policy 
with CSRS and review Project Outlines. 

Regional Associations of Directors . The fifty-three state experiment 
stations have been grouped into four regions: North Central, Northeastern, 
Southern, and Western. The directors of all thpi stations in a regional 
constitute a Regional Association. Each Association meets three times 
annually to conduct business. The Regional Association hires an executive 
director and other regional office staff as needed. The Association also 
runs the Regional Research Committee. 

Regional Research Committees . The Regional Research Committee is a 
subcommittee of three or four of the Directors in the Regional Association 
of Directors. 

Technical Committees . Membership includes the administrative advisor, 
a technical representative from each cooperating station (appointed by the 
station director), a CSRS program director, plus other consultants and 
representatives of other agencies as needed. Stations with disproportionate 
shares of the workload may have multiple representatives on the Committee. 
To the extent possible, representatives are scientists who will be involved 
in the project. 
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Adminis trat i ve Advls ors . The executive direction of a regional 
research project is delegated to a research adioinis tr&tor of one of the 
stations in the region by the Regional Association of Directors. This 
administrative advisor plays two roles; his permanent job as research 
administrator in a station, and his temporary job as administrative 
advisor. In the first role, he reports to his station director. In the 
second role, he reports to the Regional ASv«?ociation of Directors. No 
shift in physical location is usually required in taking on the temporary 
assignii'ient. 

SPECIAL RESEARCH PROJECTS PAMDIGM 
Program Planning 

Because the planning and initial development of regional research 
is ponderously slow, a third mechanism has been developed by CSRS to 
mount timely responses to sudden problems. A good example of such a 
problem is corn blight, which within two years: came from obscurity to 
threaten the U.S. com crop. CSRS's method of meeting such challenges is 
to mobilize the latent talent in the experiment stations through managerial 
action at the federal level. 

When CSRS detects a problem likely to reach national proportions 
with a year or two's time, it goes to the Congress for a special appro- 
priation. The special appropriation is sought when (1) time* is short 
before the problem becomes critical, and (2) the problem affects many, 
but not all, states, and (3) some of the expertise needed to solve the 
problem resides in places other than experiment stations, for example, 
schools of veterinary medicine or home economics. 

The means for managing these special appropriiations has hot converged 
to a uniform paradigm, so the corn blight example will be described. It 
embodies most of the principles CSRS feels are important in managing these 
special research projects. In the past, CSRS adopted a passive stance, 
merely announcing the availability of money and a willingness to consider 
proposals. Now, the importance of setting priorities and planning at the 
national level before money is awarded is recognized as crucial to success. 
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The managerial problem is to be open enough to admit all the potentially 
good ideas for solving the right problem, but not so open as to be inun- 
dated with proposals. In the com blight case, the method of management 
was the following: 

One CSRS program director was assigned full-time responsibility for 
the planning and development of the corn blight effort. During the first 
six months after the problem was first detected, this program director 
spent a major- portion of his time studying what was known about com 
blight and what the major problems seemed to be. At the same time, 
CSRS leadership was working to obtain the necessary special appropriation 
from Congress, and using the program director as a consultant on budget 
needs and justification. 

Invitations went out to the stations and other institutions with 
experti to participate in a conference on the state of research 
knowledge about the problem to be held two months after the planning 
effort began. Representatives appointed by the directors of each of 
the participating institutions presented papers summarizing what was 
knov7n in their field and then divided iiato groups to discuss which 
problems should receive top research priorities. The state experiment 
stations were encouraged to continue studying the research priorities 
question after the conference. 

Four months after the planning effort began another conference was 
held to focus on the question of research priorities — what are the 
most important problems to solve. For the first time, each state was 
asked what research it would like to perform^ 

Simultaneously with CSRS, the intramural Agricultural Research 
Service was studying research priorities, too. They asked their 
laboratories similar questions: \fliat are the most important problems 
and what contribution would your laboratory be most able to make? 

Program Development 

When it became apparent that Congress would appropriate funds, the 
program director formally asked each station and some other institutions 
to send CSRS two lists: the most important research tasks from a 
national perspective and those tasks the station was most able and willing 
: » undertake. Both lists were to be in order cf priority. 
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The program director reviewed these lists and the contribution each 
applicant had made in the planning sessions. Ability to think clearly 
about priorities, number of good ideas, and capability to produce quickly 
were the prime criteria of evalu:^tion. The program director then invited 
the directors of the 15 highest ranking institutions to Wtishington for a 
final planning session. 

All the priority and wish lists submitted were presented to the 
assembled directors in a form that enabled them to "draw lines across 
common areas of concern." After some discussion, the research program 
was reduced to the five most important kinds of research. The directors 
were then asked how they would allocate $1 million and $2 million to 
these areas should it become available. 

The result of this conference was shared with ARS , and compared with 
their priorities. A list of the projects their labs would like to under- 
take if money became available was compiled. The experiment station 
program and regional research projects were surveyed to see what current 
efforts could easily be converted to the corn blight effort. 

The program director, in consultation with CSRS supervisory staff, 
then selected nine sites to do work in the five areas. The sites were 
chosen using the same criteria as before: proven ability to react 
capably in a short time and production of many good ideas in the 
priorities-setting phase of planning. Each of the nine sites was 
requested to submit a proposal indicating what they would like to do in 
their assigned area. 

VThen the money became available, each site was told to proceed with 
their proposed work. During the period of research performance, the program 
director has kept in frequent, almost weekly, contact with the performers. 
He has checked on progress with respect to objectives, and made sure that 
the results obtained by one performer are transmitted to all the others. 

Program Evaluation 

No mechanism for program evaluation has been established. 
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ORGANIZATION 

Structural Relationships 

Performers are drawn from existing institutions on a temporary basis 
so organizational relationships change with each project. 

Staff Relationships 

Researchers . Many of the performers on special projects are station 
researchers who defer work on a formula-funded or regional project, and 
spend at least part time on the special project. Others are not, sotae 
coming from veterinary schools, home economics schools, and non-profit 
research institutes. 
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III. NATIONAL INSTITUTE OF MENTAL HEALTH 

OVTRVIEW 

The National Institute of Mental Health (NIMH) was authorized as 
an institute of NIH in 1946. The NIMH was to have an organizational 
structure similar to that of the other institutes, with authority to 
pay grants residing in an advisory council and a director reporting to 
the' Director of NIH. But, unlike the other institutes, NIMH could 
operate clinics and treatment centers, run demonstrations, train prac- 
titioners, and directly aid the States in providing mental health care. 
The Institute's charter was broader, too. It was supposed to deal not 
only with the pathologies of mental illness, but also with mental 
health as a state of community well-being. Tliese objectives allowed 
a much wider range of activities than any research agency had at that 
time, and are reflected in the agency's program today. 

In spite of its service-oriented objectives, the new Institute was 
a part of NIH and, therefore, operated in a research-oriented environ- 
ment. The Nn-Di's Congressional authorization committees were one ele- 
ment of this environment, and they believed strongly that basic research 
was the shortest cut to better treatment. Because of this research- 
oriented environment, th'.' fraction of NIMH budget devoted to $/'^rvice- 
oriented activities declined in the years just after its inception. 

In 19^8, 80% of the $6 million budget was service-oriented (service 
plus training expenditures); and 10% went to research, a ratio inherited 
from the Public Health Service's Division of Mental Hygiene, the agency 
that NIMH replaced. By 1955, research had risen to 4 7% of a $1A million 
budget, while service-oriented expenditures fell to 52% of the total. 
The portion of service-oriented expenditures allotted to training rose 
substantially from 22% of the total budget in 19A8 to 33% in 1955. 

An effort to plan a greatly expanded and balanced program for KIMH 
was begun in the early 1950s at the urging of a member of the ti'iational 
Advisory Mental !Iealth Council. The plan was d / by NlMH's Executive 
Staff and called for expenditures of $50 million within 5 years. 
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Agreement was reached with the Council that the plan should reflect two 
prime objectives: 

(1) reducing the severity of hcntal <ilisorder through provision 
of treatment and prevention programs, and 

(2) improving mental health by increasing basic knowledge and its 
utilization. 

The goal of a $50 million expenditure was reached in 1959» but the allo- 
cation to treatment and prevention activities stayed small; less than $5 
million or 5X of the total. Total service-oriented expenditures were 
not as small, however, since $20 million or AOZ of the total was spent 
on training programs. 

The Mental Retardation Facilities and Community Mental Health Center 
Act of 1963 marked a turning point in the balance between services and 
research in NIMIl. This Act provided grants for constructing and staff- 
ing community mental health facilities across the country, an authority 
that NIMH had not previously had. Because of this act, service-oriented 
expenditures as a fraction of NIMH's budget rose dramatically, so that 
by 1970, $110 million was spent on research, $122 itiillion on training, 
and $117 million on service. 

In addition to increasing the emphasis on service in the agency, 
^he Community Mental Health Center's program signaled other major changes 
in NIMH. An Institute with a large service-oriented component did not 
match well with the traditions and programs of the other institutes and 
suddenly the budget of this Institute was the largest in NIH. In 1966 
it was decided to move NIMH out from under the Directorship of NIH to 
equal status in the HEW hierarchy. 

Simul t .^incous 1 y with NIMH's move, its organization was changed to 
emphasize a commitment to social problems in mental health, a priority 
stated in its authorizing legislation. Prior to this change, NIMH was 
organized like most NIH institutes. The major branches were research, 
training, and service; and except for the service branch, those branches 
operated just as their counterparts in NIH did. They relied largely on 
unsolicited proposals for program direction, used Dual Review, and were 
relatively isolated from each other. The research branch supported 
research In the disciplines and did not seek to solve problems uncovered 
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by the service brnnch. Servi ce-orlent:cd problems were studied by the 
service branch's own research group. The training branch was largely 
concerned with meeting the needs for psychiatrists, psychiatric nurses, 
and social workers and other practitioners. 

NI^QI CTT.phasizcd its commitinent to the broader social mental health 
problems at the time of its move by adding a new division, the Division 
of Special Mental Health Programs. 

The branches of this division were the Center for Alcoholism and 
Alcohol Abuse, the Ceuzcr for Narcotics and Drug Abuse, the Center for 
Suicide Prevention, and the Center for Metropolitan Studies. Since the 
service division was already working on these problems, much of the 
first budget and staff of these new branches came from NIKH's service 
division. 

TTiese new branches differed from the old ones in two important ways. 
Each was authorized to support research, training, intramural research, 
and in some cases service. Service could include delivery of care or 
advice on how to deliver care. The branch could employ whatever means 
appeared to be the best way to solve a problem. Another difference was 
that each branch was assigned strong responsibility for developing a co- 
ordinated program directed to the fiolution of real problems. Projects 
were to be focused on selected problems and research, training, and 
sometimes intramural and service activities were to be coordinated. 
Two types of centers will be distinguished. Those which support research, 
fellowships, and training wiM be called Funded Centers, and those which 
support research, training, intramural research, and service will be 
called Total Centers. 

While NIMIl's move out from under NIH was in the planning stages, 
NIMii's extramural research branch realized it would soon be in an agency 
where the budget for services was larger than the budget for le^search, a 
strong reversal of the environment in NIH. It was able to convince 
enough people that the situation would be intolerable to gain indepen- 
dent bureau status in the HCW hierarchy at the same time that NIMH sep- 
arated from NIH. However, NIl^i*s reorganization coincided with Secre- 
tary of HKW Gardner's efforts to centralize HEW management. As part of 
this effort, the NIMH recovered its former research branch £ind made it 
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the Division of Research. The research branch had been independent for 
only six months. 

In an effort to bind NIMH's divisions more closely together and co- 
ordinate their efforts in solving priority social problems, then Director 
Yolles created a third type of center, which will be called a Cocrdinat'- 
ing Center. The first ones establislieu were the Center for Crime and 
Delinquency and the Center for Child and Family Health. Tliese centers 
were responsible for building a coherent program in their area by stim- 
ulating and coordinating proposals. These proposals, however, would be 
reviewed and funded by other branches in the Research, Training, and 
Service Divisions. The coordinating centers were a way of "trying ot*t" 
a program to see how big the problem was, whether or not results could 
be obtained, and how much of a constituency would form without the neces- 
sity of institutionalizing the problem by making it a line item in the 
budget. At the same time, a coordinated effort to solve the problem 
would be going forward. 

The next stage of structural change in NII<H, if current trends con- 
tinue, seems to be that some of the problem-oriented centers will rise 
first to division-level status, and then to institute status within 
NIMH. The alcoholism program has already achieved institute status, 
which means it has an advisoty council and its own budget. The coordin- 
ating centers are exhibiting a similar progression, except that they 
must first become Funded Centers, as Crime and Delinquency has done. 

The extrapolation of this trend is to an NIMH composed of several 
problem^or iented institutes, each with research, training, and service 
capability. Under this projection the Research Division, the Training 
Division, and the Serv' • Division would be gone, and the transition 
from an institute of Nl/i to a parallel agency composed of institutes 
would be complete. 

As this brief history indicates, there is no dominant management 
paradigm in NIMil, but rather many paradigms. In an agency that has 
undergone so many changes in organization so rapidly, and that has such 
a broad charter, this might be expected. DiscussicRi here will be 
limited to the subset of paradigms used for managing practice-oriented 
R&D. Paradigms for deliveri^ig services, conducting fundamental researcli. 
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training, and constructing facilities arc all a part of NIhQi, but will 
not be treated except for the ways in which they interact with practice- 
oriented R&D managCTTient paradigms. 

Four specific paradigms will be described: Applied Research, Coot- 
dimtinj Ccr.ter, }\<Kded Center, and Services FSD. The first paradigm 
is the sane method NIMH uses to manage fundamental research translated 
to manage problem-oriented research. Tlius , in describing the nnnage- 
ment of applied research, th« management of fundamental research is 
described ns well. The Coordinating Center paradigm emphasizes linking 
project activity in different divisions. Many of the projects managed 
by the Applied Research Branch, for example, were stimulated originally 
by Coordinating Centers. The Funded Center is a coordinating center 
with its own budget. The Services R&D paradigm is a rarity among all 
the paradigms discussed to this point, in that many of its features are 
the product of research on how to manage R&D. It is employed by a branch 
in the Services Division that is responsible for improving the delivery 
of mental health services through research on better treatment and man- 
agement methods. 

The general features of NIMli's practice-oriented R&D activity are 
the following: 

o Support is given to individuals, not to institutions, to work 

cn a problem they have proposed or been stimulated to propose 

by program management, 
o Support is awarded for a limited period of time, but can be 

renewed by competing as a new application, 
o Each proposal is submitted to a panel of the applicant's peers 

for evaluation. 

o Four different approaches to management are utilized simultane- 
ously: Applied Research, Coordinating Centers, Funded Centers, 
and Services R&D. 
o The greatest managerial effort is appl ied to : 

evaluating proposals in Applied Research and Funded Centers » 
stimulating and coordinating research in Coordinating 
Centers, and 

^ - a&scssing needs for research and stimulating adoption of 

ERJC results in Services R&D. 
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SUMM \RY OF NIMH PARADIGMS 

APPLIED 

General Characteristics 
Primary output: 

Mechanism of support: 

Managerial emphasis: 
Staffing plan: 

Program Planning 

Sources of new program 
ideas ; 

Mechanisms for planning: 
Coordination: 

ERLC 



RESEARCH 



Evaluations of new models for mental 
health treatment, techniques or de- 
vices useful in treatment, or know- 
ledge useful in planning better mental 
health services. 

Finite duration project awards are 
made to researchers or practitioners. 
These performers may be employed in 
a university, hospital, treatment 
facility, or other agency. 

Evaluation of proposals receives the 
most managerial emphasis. 

Program managers serve full-time In 

i 

their position. Review panels are 
multi-disciplinary and generally 
include practitioners. 



Because the projects supported cover 
such a diversity of disciplines and 
problems, it is difficult to identify 
specific programs in the aggregate of 
activity supported. 

Except on an ad hoc basis, no 
planning is attempted. 

Areas of interest are coordinated with 
other agencies through contact at the 
program director's level. 
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Proferam Development 



Sources of project 
ideas : 



Means of proposal 
review: 



Allocation of budget; 



Monitoring of 
performance : 

Evaluation of 
outcomes : 



Almost all project ideas are gener- 
ated outside the program management 
process . 

Every applicant's proposal is scored 
by a panel of peers and then rechecked 
at a higher level by an advisory 
council. 

Grants are paid in the order determined 
by the panel's scores. 

Substantive progress on projects is not 
closely monitored. 

Records of project output are kept so 
that the panel can be informed of an 
applicant's record in completing work 
and its quality. 



Program Evaluat ion 

Mechanism of evaluation; 



No formal procedures are used. 
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COORDINATING CENTERS 



General Characteristics 



Primary output: 



Mechanism of support: 
Managerial emphasis : 

Staffing plan: 



Additions to fundamental knowledge 
about basic processes in a specific 
problem area, solutions to problems 
faced by practitioners treating this 
problem, and dissemination of informa- 
tion about this knowledge and these 
solutions. 

No direct financial support is given. 

Determining research priorities and 
stimulating grant applications receives 
the most managerial effort. 

The Center director is equivalent in 
stature to a program director and 
serves full-time in his position. 
He may be assisted by several other 
program managers. 



Program Planning 

Sources of new program 
ideas : 

Mechanisms for planning: 



Coordination : 



A finite list of program priorities is 
established by the Center director. 

The Center director's selection of 
priorities is strongly influenced by 
the results of structured planning 
conferences. Conference invitees 
represent a cross section of the best 
people in research, training, and 
service. 

One of the Center's main functions is 
to coordinate the research, training. 
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and service activities sponsored by 
the Institute. This is done through 
discussions with other program 
directors. 



Program Development 

Sources of project 
ideas : 



Means of proposal 
review: 



Allocation of budget; 



Monitoring of 
performance: 

Evaluation of 
outcomes : 



Workshops on methodological problems, 
and the Center director himself are 
the principal sources of new project 
ideas generated by the Center. 

All applications in the Center's area 
of responsibility are reviewed by 
panels in other NIMH branches. Most 
applications are unsolicited. 

Approved grants are intermixed with 
all other grants ranked by that panel 
and paid in rank order. 

Grants in priority areas may be closely 
monitored, but most are not. 

Riecords of project output are kept so 
that the panel can be informed of an 
applicant's record in completing work 
and its quality. 
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FUNDED CENTER 

General Characteristics 

Primary output: : Solutions to problems faced by prac- 

titioners delivering rehabilitation 
services, and assistance in training 
implementors of those solutions . know- 
ledge of underlying factors causing 
problem is also produced. 

Mechanism of support: Finite duration project awards are 

made to researchers or practitioners. 
These performers may be employed in a 
university, hospital, treatment facility, 
or other agency. 

Managerial emphasis: The principal managerial effort is 

devoted to working with the review 
committee to get agreement on the ob- 
jectives to be followed in ranking 
proposals and reviewing proposals. 

Staffing plan: Program managers serve full-time in 

their position. Review panels are 
scientifically oriented, but multi- 
disciplinary. 



Program Planning 



ERLC 



Sources of new program 
ideas : 



Mechanisms for planning: 



Program activity is clustered around 
selected program objectives. New 
program ideas grow mostly out of 
successes of unsolicited projects. 

A set of program objectives are 
written by the program staff and used 
as a point of discussion with the 
review panel. 
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Coordination: Areas of interest are coordinated 

with other agencies through contact 
at the program director's level. 
Staff members also serve on technical 
committees in other agencies. 



Program Development 



Sources of project 
ideas : 



Means of proposal 
review: 



Allocation of budget; 



Monitoring of 
performance: 

Evaluation of 
outcomes : 



A good portion of the new project ideas 
are generated e.xternally to the program 
management process ; however , success 
on one project may suggest a sequence 
of research, development, and innova- 
tion activity to expand its impact. 

Every applicant's proposal is scored 
by a panel of peers, and then rechecked 
at a higher level by an advisory 
council. Program staff spends consider- 
able time working with the panel on 
program issues. 

Grants are paid in the order determined 
by the panel's scores. 

Substantive progress on proposals is 
not closely monitored. 

Records of project output are kept so 
that the panel can be informed of 
applicant's record in completing pro- 
posed work and its quality. 



Program Evaluation 



Mechanisms of 
evaluation : 



No formal procedures are used. 



General Characteristics 
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SERVICES R&D 



Primary output: 



Mechanism of support; 



Managerial emphasis: 



Staffing plan: 



Monitoring of 
performance: 



Evaluation of 
outcomes : 



ERLC 



Solutions to problems faced by 
practitioners and administrators in 
delivering mental health services. 

Finite duration project awards are 
made mostly to practitioners or ad- 
ministrators. These performers may 
be employed in a hospital, treatment 
facility, or other agency. 

An organizational change approach to 
problem solving is used. The primary 
phases of this approach are first, 
determination of perceived needs and 
potential for change; second, search 
for workable solutions; and third, 
diffusion of the results . 

Program managers serve full-time in 
their positions. Some of this time 
is allotted to staff development 
activities . 

At the time his application is sub- 
mitted, and six months after his 
project begins,. each grantee receives 
a form stating questions about results 
that will have to be answered at the 
termination date. Each grantee is 
site-visited every eight to nine months 
by his program director. 

The branch staff ranks completed pro^- 
ject on a four-point scale. The result 
is primarily used to evaluate the 
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pcrformance of the program director 
in project monitoring. 



Program Evaluation 

Mechanism of 
evaluation: 



Timing of 
evaluation ; 



Each program director sets specific 
work objectives and a range of attain- 
ment levels . Progress against these 
criteria is checked. 

The program director discusses his 
performance with the branch chief 
annually. Interim progress is dis- 
cussed in biweekly staff meetings. 



ERIC 
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APPLIE D Ri:SEARCH ACTIVITIES 

Program Planning 

The Research Division of NIMH consists of four major branches: 
Brhavioral Sciences Research, Clinical Research, Psychcpharnscolcgy 
Research, and Applied Research, 

The Applied Research Branch primarily supports three kinds of pro- 
j,ects; those that evaluate new treatment models; those that develop 
specific devices or techniques for treatment; and those that produce 
knowledge useful in planning better services. Unlike projects supported 
in the rest of the Research Division, Applied Research's projects are 
supposed to produce results that will be immediately useful to a prac- 
titioner, either in making a decision or providing a service* The other 
branches in the Research Division support work primarily in the disci- 
plines, 

A wide range of services and problems are covered. Among the topics 
are problems of the aged, child disorders, such as hyperactivity, mental 
retardation, counseling, family structure and educational development. 
Proposals come from university researchers and from practitioners* who 
want to test an idea that seems to he working. Applicants come from a 
wide variety of disciplinary backgrounds. The list of active projects 
at any one time covers so many problems, and comes from &o many dis cip^^ines , 
and is so fractured that each program director must, in general, deal 
with a very heterogeneous collection of projects. The Applied Research 
program direction generally handles a much broader array of subjects 
than his counterpart managing basic research activity. Examples from 
the list of projects Applied Research has supported are the following: 
o an evaluation of a weekend hospital built around group 
therapy 

o development of a c^>inpetence index which allows staff to 
know when patients are able to get along in everyday life 

o researcli on characteristic personality patterns of chronic 
wanderers 

o research on how individuals cope witli natural disasters 
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o research on the way Itnmigranc minorities find a place to 
stay in a new city. 

Most project proposals received by Applied Research are unsolicited^ 
or h.ive been stimulated by one of the Coordinating Centers, but occasion- 
ally an issue will arise th.it secr^^i important to a program director. 
VThcn this h.ippcns, a planning effort m.iy be mounted. One example occurred 
after the first outbre.iks of urban violence several years ago. A pro- 
graiTi officer became concerned over the apparent lack ol kncrwlcdgc about 
what interventions would reduce the amount of violence in riot situations. 
A spocial panel of researchers active in the field of violent behavior 
was convened to assess what research projects were needed. Using the 
panel's re comrrcndat i ons , the st«ff decided on a series of projects. 
The staff then stimulated a number of research proposals. Tl^e proposals 
were evaluated using another ad hoc panel of experts, since the regular 
panels were not felt to be expert enough in the subject matter of the 
proposals. In general, however, program planning is not a frequent 
activity in Applied F.csearch. 

Activity in Applied Research is divided into two program areas: 
Juvenile Problem.s and Social ProUlems. Each area has a review panel 
and a program director. Most of the time these principals engage in 
Krogran Development activities as described below. 

Prog r am Development 

In Applied Research, Program Development is conducted with a method 
that borrc'ws from both Single Review and Dual RevieWiv It is like Single 
P^view in that the program director is the one w^o runs the primary re- 
view panel. It is like Dual Review in that the primary panel reviews 
and scores every proposal and that there is a second level of review by the 
National Advisory Mental Health Council. 

Instead of stimulating research proposals in Applied Research, the 
program director spends a great deal of time assisting applicants in 
the preparation of proposals. Because many come from applicants closer 
:o the practitioner's world than the scientist's world, and research on 
real problems is not often "clean," many proposals received are 
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TnctJ)odolo;;i cai 1 y weak or poorly written. The underlying ideas are 
sound, but the test procedures and research design are deficient. The 
program directors knew they can have a substantial effect on the propos- 
al's chances of success by pointing out methodologic£>l deficiencies, 
styles that appeal, and background inforaation needed. By the nature 
of the applicant population in Applied Research, the progran director's 
influence on the review process is greater than in fundamental research. 

Tlie proposal review process starts in NIH's DRG where referral 
officers route proposals to NIKll's referral office, which routes them 
to one of the two panels in Applied Research. The program director of 
the panel to which a proposal is assigned selects a primary and secondary 
reviewer with an instruction to request any additional information needed 
to rryj^]etc an adequate written review. If enough extra information Is 
needed, the program director arranges a site visit. 

The review panel meets three times annually to review about 30 to 
AO proposals in a three-day session. Each proposal is discussed In turn 
for a half-hour and then approved or disapprc^ved by the panel. Discus- 
sion is led by the primary and secondary reviewers. Proposals approved 
by a majority of the panelists are then scored by each panelist on a 
scale of one to five by secret ballot. The propoisal's score Is the 
average of the individual scores. 

The program director then writes a summary (pink sheet) of the 
proposal and the panel's comments for presentation to the National 
Advisory Mental IJcaltli Council. Tlie Council meets and reviews the 
summaries of all projects pending in all the branches of NIMH. Due 
to the extreme breadth of topics treated by NIMH, the Council can focus 
Its attention only on selected grants. The rest are approved "en bloc." 

The Council's attention is keyed by comments from the program 
directors. The program directors will point out proposals thought to 
liavc received inadequat<f review, those thought to be especially program 
relevant but low rated, ^r ones disapproved by more than one reviewer. 
The Council then notes a change in score or project budget, or refers 
the proposal back to the review panel for another look. The procedure Is 
virtually the same* as occurs in NIH Council meetings. 
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After the Counci 1 ' s act ion , each program director interdigitates 
the proposals ranked by his conmittcc and then pays dovn the list until 
his budget is exhausted. The size of his budget is dcterminc»d by the 
Applied Research Branch Chief and the Director of the Research Division. 
Project nonitoring Ir, rondijcted no differently than in Dual Review. 
However, in distinction from Dual Review, copies of all papers and re- 
poits written by a grantre are kept and provided to the panel with the 
gr;3ntee's next application. The panel considers the quality of these 
outputs in ranking the new proposal. A flow chart of this process ap- 
pears bclcw. 

A key difference between this paradigm and Dual Review is that the 
program director nominates the panels. How this is done and the philoso- 
phy folJoved is described in the Staffing Rel/itionship section. A dia- 
gram of the Program Development process follows in Figure 6* 

QRGA.N1ZAT1QN 

_St ructural Re 1 at ion^liip s 

A chart of the ^^IMH organizations appears in Figure 7. 

Staf fing Relationships 

Program Directors . Vith rare exception, all program directors 
have M.D. or Ph.D. degrees and usually some experience as a bench sci- 
entist. There is a small level of migration from intlamural research 
to program management, but none in the reverse direction. Program 
directors are selected by the Applied Research Branch Chief and the 
Director of Exlramural Research, and have responsibility for a single 
program area. Virtually none serve another role besides their assigned 
responsibi] * ty . 

Review Panel Members . The program director serves as co-chairman 
of the review panel and recommends replacements for panel members from 
the community of scientists when their fci^r— year t^rm expires. These 
nominations are then approved up the line to the Secretary of HEW, but 
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seldom denied. The review panel co-chairman is chosen by the program 
director. 

Since the range of disciplines and problems covered by each panel 
is large, nominating panel members is a difficult job for the program 
director. In general, the best policy is to select two or tliree panel- 
ists from each discipline frequently relevant to the program, and at 
least two or more practitioners. It is especially important to have 
two or more extremely strong methodologists on the panel. 

Nation al Adv isor y Mental Health Council Members . At least six are 
required by law to be authorities in mental health or science fields 
important to the Institute's concern. Two are ex officio representatives, 
one required from DoD and the other from the Veterans' Administration — 
and as selected, they tend to be scientists. Of the remaining six posi- 
tions, one or two arc usually lay representatives and the rest scientific. 
Advisory Council members are nominated by the Institute Director and 
approved up the line to the Secret; of HEW. 



-60- 



COORDINATING CENTER ACTIVITIES 
Program Planning 

Coordinating Centers are a distinctive form of organizational unit 
that NIMH has devised for coordinating R&D projects in a particular 
problem area that are being performed in its separate divisions. 

Over the, last few years, Coordinating Centers have been organized 
around a variety of problems and in a variety of ways. The section on 
the aged, which is not formally designated as a Coordinating Center, 
is a group of program managers in the Special Mental Health Programs 
Division. The Center on Child and Family Mental Health is designated 
as a Coordinating Center in the Special Mental Health Programs Division 
and consists of seven people. The Center on Schizophrenia is a part 
of the Clinical Research Branch of the Extramural Research Division 
and is run by two people, a Director and his assistant. While still a 
Coordinating Center in the Special Mental Health Programs Division, the 
Center on Crime and Delinquency was run by two people, the Center Direc 
tor and his assistant. The Crime Center has now graduated to Funded 
Center status in the Special Mental Health Program Division. 

Even though each Coordinating Center runs a little differently and 
has different relationships with other NIMH divisions, some character- 
istics are uniform. 

o All Coordinating Centers keep track of projects in all three 

functional divisions; Research, Training, and Service, 
o None of the Coordinating Centers run proposal review committees 
Grants stimulated by a Center are reviewed by the permanent 
panels run by other branches, 
o The research areas being dealt with are vaguely defined, and 

not yet very effective in solving practical problems* 
o A substantial amount of research, training, and service grant 
activity related to the problem was present when the Center was 
formed (e.g. ^. 30% of the extramural research budget was for 
s chizophrenia. 
Some differences among the centers are that: 
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o Some centers deal with a problem that is generally accepted 

to be a mental health problem (e.g., schizophrenia) while 

other centers deal with a problem on which other agencies are 

also doing research, 
o Some centers dej 1 with a comparatively narrowly defined problem 

(e.g, schizophrenia), while other centers have a domain that 

covers a tremendous range of problems, 
o And, as already mentioned, some centers are in functional 

Divisions, while some are in the Special Mental Health Proprams 

Division. 

The level of frustration among the leadership of these centers over their 
ability to influence project activity varies. It correlates with the 
factors listed above. Correlation, however, does not prove causation. 
The Director of the Schizophrenia Center, which deals with a more narrow- 
ly defined, primarily mental problem, expresses less frustration with 
his position than the other center directors. The managers interviewed 
in other Coordinating Centers expressed a great deal more frustration 
over their asserted inability to affect R&D activity. In each case, 
these centers deal with a broadly defined problem that is not necessarily 
a mental health problem. Most of the frustration stems from territorial 
issues inherent in the center's lack of a budget. 

Centers have been started for a variety of reasons in addition to 
the desire to coordinate R&D activity. One reason is the presence of 
other centers. When a center is started, it gives visibility to a 
problem area within an agency. Strong constituencies, which have little 
interest in being left out, bring pressure to form a center for their 
problem. Another factor that is associated with the * otablishment of 
centers is the creation of an interagency or presidential level commission 
on a problem in which Nll^ffl is interested. As in the case of community 
mental health centers, large programs sometimes result from these com- 
missions. The Coordinating Center is a way for NIMH to indicate active 
interest in a prohletn v?ithout becoming committed to a line item in the 
budget . 



•62- 



Coordinating Centers may be a very appropriate mechanism when only 
rudimentary knowledge about a problem is available. By establishing 
a Coordinating Center instead of a permanent unit with its own budget, 
there is reduced chance that one clientele group or one perspective will 
dominate in solving the problem. Since it cannot fund projects directly, 
the Center must work cooperatively with other Centers. And, to be ef- 
fective in its role, the Center must work with more than one branch, or 
be charged with territorial infringement. In being forced to work 
cooperatively with several other branches, the Coordinating Center is 
naturally kept open to new ideas and opportunities. The Center for 
Studies of Schizophrenia is a good example. Even though schizophrenia 
has been studied from some time, it is still poorly understood. A vari- 
ety of therapeutic methods , including drugs and behavioral approaches, 
have been tried, to some success, but the underlying mechanisms of the 
disorder, and a cure, have not been found. After years of trying to 
cure schizophrenia with drugs, there is evidence that at least some 
kinds of schizophrenia might be a social problem. No lesions have been 
observed and apparently it is not a disease. Under these circumstances, 
it is advantageous from a managerial point of view to avoid tying an 
R&D program lo one research discipline or treatment approach. Estab- 
lishing a separately funded activity for schizophrenia would have run 
the risk of overspecif ying the problem too soon. 

Since the several Coordinating Centers operate in different ways 
to a greater extent than with other management approaches, there is no 
common management paradigm. As an alternative, the operation of one of 
these Coordinating Centers will be described. 

The director of this Center has a solid reputation as a researcher 
in the problem area with which his Center is concerned. The director 
feels that this makes an important difference in his success in dealing 
with program directors and panels in other branches. Program directors 
see him as a scientist committed to solving a problem, who could just 
as well be at the research bench, and not as <^xx administrator encroach- 
ing on their territory. In addition, the program director can approach 
the review panelists as an equal, which greatly improves his impact. 
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The Center deals with program directors in all three divisions of 
NIMH and, for the most part, three review committees in those divisions* 
Two of these review committees are in Research and one is in Training, 
The Service Division is used primarily as a source of projects or oppor- 
tunities, which can be made to yield research results if a proper re- 
search design is added. There is little attempt to promote specific 
service projects. The Center director keeps track of whatis going on in 
the Training and Service Divisions by reading summaries of review panel 
actions (pink sheets) and talking with program directors. 

One characteristic problem that gets the director's attention is 
the tendency in training and service to apply uniform treatments without 
regard for categories of problems. For example, training in psychiatric 
residency programs is treatment-oriented, not disease-oriented. Thus, 
trainees do not specialize in diseases, but in treatments. The Center 
director works with program directors in the Training Division by explain 
ing what research has discovered about the disorder with which he is 
concerned and encourages them to get improvements incorporated in new 
training grants. 

Besides his own knowledge, the Center director has used structured 
conferences as an aid in developing program priorities and ideas. For 
one of these conferences, about 100 professionals from research, train- 
ing, and service were invited to assess the state of knowledge about 
the center's problem; assess how well this knowledge was being applied 
in training and service; and recommend research, training, and service 
priorities. Experts in each of these functional areas were asked to 
prepare state-of-the-art papers. In research, eight were prepared, one 
each in topics such as genetics, bio-chemistry, perception, and epidemi- 
ology. In service, consultants were hired to do a partial survey of 
facilities. In training, practitioners were hired to summarize the 
current state of affairs. 

After an opening plenary session, the conference was broken into 
eight working groups; four in research, and two each in training and 
service. Each group was assigned a portion of the papers in its func- 
tional area, and given the task of reviewing these papers and writing 
a group position paper. The next morning, these papers were presented 
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at a second plenary session. After this session, the conference was 
broken down into small groups again, except that this time each one 
contained a balance of research, training, and service people. Each 
group was assigned two or three common questions, plus a special ques- 
tion. Two of the common questions were: What are the barriers to 
utilization of research findings? and What research possibilities have 
the most promise? One special question was: Iwiat should be required 
in psychiatric training for disorder, where X was specified? The lists 
of priorities showed a lot of overlap, including some motherhood recom- 
mendations, but some surprising ones, too. One and a half years after 
this conference, a conference of the group chairmen will be held to 
assess what progress the Center had made since the first meeting, and 
to replan the Center's priorities, if necessary. 

At the present time, the Center director has three priorities under 
development. 

Program Development 

The Center director has four means of stimulating research activity 
in his priority areas. One of the most successful means are small 
workshops, where methodological issues pertaining to one of the priority 
areas are discussed. Personal visits and seminars delivered at univer- 
sities and elsewhere are a second means. The journal that the Center 
publishes is a third means. 

In each volume of this journal, the Center di; -"tor writes an 
editorial highlighting particular issues and what research he thinks is 
most important. The editorial is written in such a way that readers 
will infer what proposals would be most warmly received by the Center. 

A fourth means is by direct collaboration with another researcher. 
This technique is used when the level of research being conducted in a 
priority area is low. Sometimes the cause of low quality research is a 
general pessimism in the research community engendered by years of con- 
sistently fruitless effort. If the Center director thinks he knows a 
better way to do research in cne of these areas, he finds an extramural 
collaborator who writes a grant proposal himself and submits this 
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application through normal channels with the collaborator's and his own 
name on it. Usually, the collaborator is a practitioner who has the 
needed sample of patients or other data. If the research is successful, 
it becomes a model for doing research in the priority area. 

The Center is not authorized to evaluate proposals, so all of them 
are sent to review panels run in other branches. The mechanics of this 
review process are exactly the same as for Applied Research, which were 
described in the previous section and Figure 6; therefore, they will 
not be described in detail here. The Center's proposals compete against 
all other proposals ranked by the branch, and are interdigited with them 
in determining the order of payment. 

The Center director spends a fair amount of time working with the 
review panels ard program directors that handle his proposals. It has 
been the Center director's experience that at least two or three of the 
panelists must be recognized contributors in the center's field, or the 
Center's proposals will fare poorly in the evaluation process. Since 
panelists are appointed through a continual bargaining process between 
the program director, his panel, and the Division Director, the Center 
director must get involved, or not expect many representatives from the 
Center's field on the panel. 

The Center director is much more successful in this effort if he 
works with the panel on substantive issues. One way is to discuss his 
research program with the panel at the beginning of their review session. 
His competence as a scientist helps greatly in getting the panel's 
interest in this discussion. Another method used is to bring problems 
before the panel to get their advice. 

One problem brought to the panel was a situation where several 
researchers working in one of the Center's prio lly areas were working 
independently and using separate data bases. A workshop on the matter 
failed to stimulate the desirable collaborative effort, partly because 
none were willing to pay the extra costs involved. The panel discussed 
the problem with the Center director. They suggested that he find 
somebody who would be willing to submit a grant that proposed funds 
for a new experiment, which they specified, and for the researchers to 
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exchange lab visits and data. The panel said they would like to review 
the application. It was reviewed, received a high rating, and was paid. 
Some panel members are now participating in site visits of the ongoing 
grant . 

The Center also fulfills an information dissemination role through 
publication of the journal mentioned previously. Each issue of the jour- 
nal contains review articles and original papers on a selected theme 
along with brief summaries of negative research findings, NIMH program 
descriptions, announcements, and lists of recent books. The journal is 
sent to researchers, clinicians, teachers, and laymen and has a format 
directed to the practitioner. The journal serves as a vital link between 
research, training, and service activities in the Center's field. 

O RGANIZATION 

Structural Relationships 

An organization chart showing some of the Coordinating Centers ex- 
plicitly appears in Figure 7^ Others are found in the Research and 
the Service Divisions. 
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FUNDLD CENTt:R ACTIVITIES 
Progran Planning 

Organizationally, the Funded Center is indistinguishable from the 
Applied Research Branch, for the staff consists of 3 prcgrsrs director, 
plus assistants, and a multidiscipl inary , scientific review panel. Many 
of the procedures followed, especially in proposal review, are virtually 
the same. Functionally, however, the Funded Center's responsibility is 
different from that of the Applied Research Branch. The Director of the 
Division of Special Mental Health Programs, to whom the Centers report, 
has charged each Funded Center with mounting a coordinated, directed 
attack on problems in its area of concern. Applied Research docs not 
have a similar mandate. 

The Center's ability to mount a coordinated, directed attack on 
problems is aided by its authority to fund both research and training 
grants. This allows the Center to take advantage of solutions or know- 
ledge when they coidc along and to assist their introduction into prac- 
tice. For example, if a research project determines that a certain way 
of using citizen volunteers as probation agents for misdemeanor offend- 
ers is effective, and an evaluation project is done to verify the dis- 
covery, then a curriculum for training the volunteers needs to be devel- 
oped, and instructors for the curriculum need to be trained. Further- 
more, a research project might be indicated to determine whether the 
same approach, or one using ex-offenders would work with criiiiinal of- 
fenders. The ability to choose just the funding instruments needed to 
do a certain job is crucial to the Center's ability to program R&D pro- 
jects. 

While the Funded Center can choose between a research instrument 
and a training I'/istrument in solving a particular problem, it does not 
have much control over the aggregat balance between research and train- 
ing. This balance is determined by the N1>DI Director's office. Because 
research and training are line items in the NlI-Cl budget, the Director's 
office must constrain the Funded Center to separate accounts for research 
and training, or have trouble meeting its line item commitments. 
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A Funded Center's budget is typically not very large in relation 
to the dimensions of its problem area, so the goal of mounting coherent, 
directed programs makes it necessary to define objectives and concentrate 
resources on them. As an example, the Center for Crime and Delinquency 
spent $3.5 million on research grants and $2.1 million on training grants 
in 1970. 

Another reason the Funded Center sets program objectives is to build 
accountability into the program management process. Without objectives, 
it is impossible to measure success, and more difficult to get the sev- 
eral people involved in the Center working toward the same ends* 

DctcriT^ining program objectives has consumed 5 to 30% of the program 
staff's time over the years; however, no r'^r^ular process is followed. 
For the most part, the Center's objectives have been written by the pro- 
gram staff. Interactions with the review panel, knowledge gaps that 
have become apparent over the years, and project successes all have their 
effect on successive versions of the objectives. 

Program objectives and project activity are coordinated with other 
agencies working in the same area by periodic meetings between the pro- 
gram directors involved. Sometimes these meetings result in referral of 
applications from one agency to another. Also, Center staff members 
serve on committees in other agencies. 

The Center's objectives guide program development in two ways. 
They guide the program staff's thinking in stimulating research proposals 
from Investigators, and they provide a framework for discussing project 
priorities with the review panel. 

Program Development 

It is important that priorities be discussed with the review panel 
because their judgments essentially determine the list of projects that 
the Center supports. If the panel's priorities are not the same as the 
Center's, then the Center will have a different set of objectives than 
planned. Center management gets agreement on priorities between the 
Center and the panel by spending a good deal of time discussing priori- 
ties and objectives with the review panel. The first day of each t!*ree- 
O ^ day review session is set aside for discussion of program objective , 

EHJC 
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the significance of project outcomes, training issues, the funding 
situation, or any other issue that becomes important* 

These policy sessions are conducted informally with free-wheeling 
discussion among the panel members. The program director, vho chairs 
the mectiTig with an elected riember of the panel, modulates the course of 
discussion by interjecting comments at selected points. He attempts to 
keep discussion on the track and steers away from situations where one 
panelist dominates the discussion. To raise the quality of discussion, 
the program staff usually prepares an issue paper on the items of dis- 
cussion that sketches the current state of affairs and outlines alterna- 
tives. 

The effect of these discussions on the Program Development process 
is indirect, for votes on issues or resolutions are rarely taken. The 
attempt is to work for a consensus amiong the panelists and then rely on 
consistent application of the policy agreement whenever it needs to be 
applied. As an eaiample of how this works, one Center discussed the 
utility of funding more verbal conditioning experiments. Verbal condi- 
tioning experiments are generally tight and neat scientifically, and 
therefore attractive to some scientists, but very difficult to translate 
into effective treatments. After spending some time discussing this 
issue, the panel gave proposals to do verbal conditioning experiments a 
low rating, even though such experiments are thought by some to be "good 
science . " 

The rest of the Program Development process is similar to the Ap- 
plied Research paradigm except for some minor variations. After spend- 
ing the first day on policy, projects are brought up for reviev?. Each 
proposal is discussed in turn. Discussion is begun by the panelists 
assigned as primary and secondary reviewers. After their presentation, 
the program staff comments on the proposal, indicating the previous 
record of the applicant, whether or not he submitted final reports, sub- 
stantive comments on the proposal, and the ability of the applicant to 
administer his grant. Program staff takes care not to give the appear- 
ance that it is advocating a proposal, otherwise the panel will be 
likely to give it a low rating. In the evaluation session, the program 
staff makes it clear that the panel is responsible for making the 
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proposal evaluation decisions on the basis of scientific quality. 

One difference between tlio Center*s panel and Applied Research's 
panel is that only 15 to 20 applications are graded in the two days per 
session allotted to evaluation. This makes it more feasible for all 
panelists to read all proposals before the meeting. In addition, the 
panel grades clearly superior applications after only ten minutes, or 
less, of discussion. 

As in Applied Research, each proposal Is ranked by the panelists on 
a scale of one to five using a secret ballot. The proposal's bcore is 
the average of the individual scores. 

After being ranked by the panel, each proposal is reviewed again 
by the National Advisory Mental Health Council in exactly the same way as 
in Applied Research, The Center then pays grants in order of their score 
until the budget is exhausted. No grouping of projects by objectives is 
attempted in setting pay limits. 

Grantees are somewhat more closely monitored than in Applied Research, 
After the first year, eacli one is asked to state how his research will 
be useful in solving one of the Center's problems. The grantee also 
submits an annual progress report and may get a call from the program 
director at any time. During performance, small changes in the grant 
budget are approved by the program director, but significant ones are 
brought to the review panel. One year before the termination date of a 
grant, the program director gives the granteee his views on items that 
should be in the grantee's final report. 

The only other difference between Applied Research and the Funded 
Center is that the Center's program staff spends some, though not a major 
part, of its time stimulating research proposals. Stimulation is often 
necessary to move a successful idea from one stage of implementation to 
the next, or to investigate a particularly interesting possibility. 
This is primarily done by telephone contact, but scientific conferences 
and side trips are used, too. In a short time, the program director 
builds up a substantial list of people interested in working on his prob- 
lem* The review panel and former panelists are also strong linkages to 
potential performers, 

A diagram of the Program Development process appears in Figure 8. 
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ORGANIZATION 



Structural Relationships 

The Center is not large enough to warrant an internal structure. 
Its place in the NI^^^ hierarchy is indicated in the organization chart 
in the Applied Research paradigm. 

Staffing Relationships 

Program Directors and Assistants . With rare exception, all have the 
M.D. or Ph.D. degree and usually some experience as a bench scientist. 
There is a small level of migration from intramural research to program 
inanagement, but none in the reverse direction. Program Directors are 
selected by the Special Mental Health Programs Division Director. Virtu- 
ally none serve another lole besides their assigned responsibility. 

Review Panel Members . The program director recommends replacements 
for panel members from the community of scientists when their four-year 
term expires. These nominations are then approved up the line to the 
Secretary of HEW, but seldom denied. The review panel co-chairman is 
chosen by the program director. 

Since the range of disciplines and problems covered by each panel 
is large, nominating panel members is a difficult job for the program 
director. In general, the best policy is to select two or three panell- 
ists from each discipline frequently relevant to the program. It is 
especially important to have two or more extremely strong methodologists 
on the panel. 

National Advisory Mental Health Council Members . At least six are 
required by law to be authorities in mental health or science fields im- 
portant to the Institute's concern. Two are ex officio representatives, 
one required from DoD and the other from the Veterans' Administration — 
and as selected they tend to be scientists. Of the remaining six posi- 
tions, one or two are usually lay representat-^ves and the rest scientific. 
Advisory Council members are nominated by the Institute Director and 
approved up the line to the Secretary of HEW. 
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SERVICES R&D ACTIVITIES 
Prograra Planning 

Over the last five years, the managers currant ly runnin^^ the Mental 
Health Services Development Branch of the Division of Mental Health Ser- 
vices have evolved a strategy and a set of tactics for conducting R&D on 
methods of delivering mental health services by successively altering the 
Applied Research paradigm. The first stage was stimulated by an evalua- 
tion that showed that it was possible to obtain a written statement of 
findings friom only 40% of completed projects. The correction applied was 
careful rnonitor^ing of grantee performance. A subsequent evaluation re- 
vealed that availability of a final report did not very often ensure 
utilization of its results. Less than 20% of the principal investiga- 
tors could name one place using their results. The correction added 
this time was an inducement that encouraged each grantee to disseminate 
his final product with written publications. Studies of mental health 
innovations then showed that written materials initiate less than 9% of 
the adoptions in mental health service delivery. The third evolution 
stage vas a program of special devices to stimulate utiliz \tion of re- 
sults. The difficulty that arose this time was that not enough of the 
projects results solved problems that practitioners frequently had. 

The fourth stage of development now being instituted emphasizes an 
organizational change approach to R&D management. All the mechanics 
developed earlier are retained, but the role of the program staff is 
drastically changed from one of managing a grant program and then stimu- 
lating adoption of results to finding out what problems practitioners 
have, and then getting the technical assistance thpy need to solve their 
problems. In working with practitioners, the program staff uses what 
has been discovered in research on organizational change. Some of the 
more relevant findings used are that the probability of achieving a 
change is related to: 

o the availability and amount of resources needed to make the 
change 



-73- 



o the congruence between the adopter ' s values and those iroplicit 
in the innovation , 

o the compatibility with available information of information re- 
quired as input by the innovation, 

o the degree of dissatisfaction with conditions on the part of 
the potential adopter, 

o the political conditions, such as crises that prevail at the 
time of adoption, 

o the resistance mounted by external parties, and 

o the potential yield of the innovation. 
As an example of how these findings are used, the branch encourages 
practitioners to conduct evaluations of mental health services. The 
effect of these evaluations is often enough to heighten dissatisfaction 
with existing conditions and, therefore, to raise the demand for innova- 
tions. To aid this process, the branch has a priority R&D on improved 
methodologies for conducting evaluations. 

Another branch priority is staff development. Biweekly staff devel- 
opment meetings are held to exchange managerial, organizational, and 
sometimes interpersonal problems, and learn management skills. Courses 
and manuals for learning organizational change techniques have been de- 
veloped. A publication called the Journal of Innovations and Current 
Conclusions is produced with staff members assigned responsibility for 
input in their respective specialities. This works to maintain the pro- 
gram director's substantive expertise. 

Branch activity is loosely organized. Each program director on the 
staff has responsibility for one of 16 program areas. These program 
areas are overlapped to encourage exchange of resources, and interactive 
discussion between the program directors. The program areas are clus- 
tered into sub-groups composed of similar problem areas with each pro- 
gram director serving as sub-group coordinator on a rotating basis. 
The coordinator convenes the sub-group for mutual planning efforts, co- 
ordinates mutual planning efforts, and handles administrative chores. 

One management experiment underway at the present time is to assign 
one program director in a sub-group responsibility for assessing "emerg- 
Q ing needs" and translating them into research project ideas to be 
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considered and developed by the rest of the sub'-group. This program 
director spends essentially all of his time in a program planning mode. 

Branch priorities are set by the branch leadership and revised when 
conditions warrant . Because of the way program area responsibilities are 
assigned, priorities sometimes cut across branch sub-groups, allowing 
contributions to a priority by more than one sub-group. As an example, 
program priorities at one time were: 

(1) Children's mental health services. 

(2) Sheltered services for the mentally ill. 

(3) Evaluation and innovation in mental health. 

(4) Allied systems of services delivery. 

These can be compared to program area responsibilities listed in the 
Structural Relationships section of this paradigm. 

In setting priorities, the branch leadership relies on several 
sources. First, whenever program staff attend a meeting or conference 
they circulate a small survey form that has been designed. Many of 
.ihese are practitioners' and administrators' meetings. One result has 
been the discovery that perceptions of critical areas in mental health 
vary greatly among categories of respondents. Researchers often have 
different perceptions of problems than practitioners and administrators. 
A second method is formal program evaluations, contracted especially to 
assess impacts and availability of community mental health services. A 
third source is now in the design stage. It will be a system for collect- 
ing bioraetric data on populations in mental health facilities across the 
country. All of these methods are adjuncts to the usual political and 
social pressures. 

Program activities are coordinated with similar work in other places 
through the usual meaas : participation on task forces and informal 
contacts . 

Program Development 

Each program director is responsible s.or coordinating project activity 
in his program area along several dimensions. One responsibility is to 
have the projects active in his area reflect branch priorities and be 
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derived from specific needs felt by practitioners. In addition, there 
should be sufficient diffusion and utilization mechanisms built into 
each project. The program director is responsible for stimulating 
auxiliary diffusion and utilization mechanisms if a project turns out to 
require them. The program director is also responsible for developing 
collaborative projects on different aspects of a problem, when this is 
desirable . 

The program director can meet this responsibility because the 
branch's project workload is light compared to other paradigms. Each 
program director has about four new grants each year, and approximately 
seven active at any one time. 

Despite his responsibilities for coordinating grants, the program 
director cannot award, or even promise, grants to individuals. The pro- 
gram director is limited to stimulating grants he needs because all pro*- 
posals must be evaluated by the branch's review panel. 

He does this by finding qualified potential performers, and consult- 
ing with them in preparing a proposal. The principal means of finding 
potential performers is by working through practitioners and researchers 
dealt with over the years, or recommended by other program directors. 
Not all the proposals funded by the branch are stimulated, however. 
Many applications are received from practitioners and others who feel 
they have good solutions to a problem. 

Before investing effort in stimulating a proposal, the program 
director is responsible for searching the literatui and his contacts to 
see if a solution already exists. A search of NIMH projects is performed 
first using NIMH's computer information system. If another NIMH branch 
has already found a solution, the program director's next job is to get 
the solution adopted, not to stimulate a research projects A literature- 
screening service is also used, but in Services R&D, solutions are not 
always in the open literature. This happens because many projects are 
performed by practitioners or administrators who have neither the time 
nor the incentive to publish in journals. Articles are solicited from 
these people for the staff's Journal of Innovations and Current Conalu-- 
sions^ but this does not go far toward solving the problem. 
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Proposals are evaluated in essentially the same way as with Funded 
Centers and re-reviewed by the National Advisory Mental Health Council. 
The primary review panel meets four times a year for two or three days 
each time. One of the sessions is allotted to discussion of program 
issues, and at the rest, part of the time is reserved for discussion of 
program priorities. At the program issues session, program directors 
brief the panelists on needs in their area. 

In the future, as the panel's workload grows, an additional panel 
will be formed. One will handle research grants and the other develop- 
ment grants. 

Once a project is funded, the grantee's performance is monitored by 
the program director. He visits the grantee every eight or nine months 
to discuss progress. At every contact the program director asks the 
grantee what he expects his final results will be, and how they will be 
diffused or disseminated. The program director offers guidance in pre- 
paring a diffusion plan and the final report. The program director knows 
that the question of how results will be disseminated is important, for 
research supported by the branch has shown that investigators who plan 
for diffusion of results from the beginning of their project are much 
more likely to have their results used in practice. For this same reason, 
the branch includes a questionnaire with each application mailed to a 
prospective grantee. The questionnaire has the same questions on it 
which the grantee will be asked to answer when his project is finished. 
Some of the questions are: What information did the project yield that 
would be of value to others? By what method were results obtained? 
What dissemination efforts were completed? Who is using the results? 
The same questionnaire is sent after six months of a {project have elapsed 
to remind the grantee that he will be held accountable for dissemination 
at the end of his project. At the end of his project, the grantee is 
asked to fill out the questionnaire. By this technique, the percentage 
of projects submitting final reports has increased from 40% to 95%, and 
the number of projects reporting adoption of results by others has risen 
from 19% to 50%. 

The program directors are in a position to counsel grantees on the 
Q best ways to diffuse project results, since over the years, research on 
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this subject has been done by the branch. A set of proven models is 
available for use in specific situations. The branch can also facili-. 
tate adoption directly because it has budget to fund demonstrations, 
training activities, and some other utilization aids in addition to 
research and development. 

A description of the program development process appears in 
Figure 9. 

Program Evaluation 

Instead of evaluating its programs directly, the branch evaluates 
each program director's performance using a "Goal Attainment Scale" 
method. At the beginning of each year, the program director, in consul- 
tation with the branch chief and the branch staff, decides on a set 
(usually about five) of specific work objectives for the ensuing year. 
For each objective, five levels of achievement, in terms of specific 
outcomes, are written down. The levels are assigned integer values from 
-2 to +2 for the highest to lowest levels of attainment. Each objective 
is then assigned a weight from 0 to 10. At any point in time, program 
director's "attainment score" is simply the product of each objective's 
weight factor times its level of attainment, sumiTied over all objectives. 

The program directors and the branch leadership revise these rating 
grids quarterly, but progress is evaluated by the branch chief once a 
year. Progress toward each program director's objectives is one of the 
items discussed at the biweekly staff meeting. By discussing progress 
in joint session, self-deluding estimates are avoided. 

ORGANIZATION 

Structural Relationships 

An or inization chart for NIMH appeared in Figure 7, and shows the 
position of Mental Health Services R&D branch in NIMH. A diagram of the 
inside organization of this branch appears in Figure 10. 
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Activity 



tetctrch needs, 
•Xnthcflft of nev 
Approachira, ttlvu' 
Ittlont of new 
propctfta. 



Evaluation of 
proposals (Srudy 
section Bcets 2 
CIms per year 
for 2 or 3 days. 
At review 
sessions it 
rsnks IS to 20 
proposals • The 
Council Dcecs 6 
vtska l«ter, 3 
tlM ^ per year.] 



Allocation of 
Institute budget 
to projects. 



Koflltorlng and 
suppor t Xng 
ongoing projects. 



Evaluation of 
pr^ect outcooei. 



I»plc»entatlon 
of results. 



>< 
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Researchers 



Referral 
_Offlcers 



yiRurt ^ ; Prosraa Ocvclopnent for Services %kO 



Frotraa Director 



Rcvlct/ P.incl hcnbers 
_ilO to 16 per parcl) 



Advisory Council 
Members (1< | 



Send proposal 1 
toNIhOi. ( 



Sort incoming 
proposals to 
prograa 
df reccors . 



If proposal turned 
dowu. take cocnents of 
prograr! dlrcctur (If 
offered) and tcvIsc 
■PpUcntlon. Then 
resut>slt. 



o ContscC pctentlsl grantees to see 

If they vlll do needed rcsesrch. 
o At conferences dlstrlbut« sMfvcy 

fora on needs, 
o Suggest fbprovencnts In prellol- 

esry proposals, 
o Talk to practitioners at neetlngs 

end individually. 



o Keet once a year for 2 days Co 
discuss progrAB Issues. Dlscuxslon 
ltd by prograa directors. 



Request thst researchers send 
notice of proposal submlssl««i. 
Vatch Its progress through 
sorting ptocesi. 



.» o Assign Incoming proposals ^o one. ] [©Prepare writer' -luoaar'^s of 
to-three psncliits for preparatlon^J assigned proposals. Kc^d *» 
of vritten review before the panel I many of the Teit as tins is 
I fceeting. | available. 



0 Rate Institute's per* 
forviancc using stand.-<rd 
fopB. Issued by 
prograa directors. 

o Investigators arc required 
to Ir^ludc a tentative plan 
for diffusing project ri.*Bulta 
fn the project proposal. 

o Investigators arc cicpoctcd to 
take socte action toward 
dlsseolnaLlng or dlffunlng 
the results of their project. 

o Supplementary utlJlaiatlon 
trants can be stlculoled 
If needed. 



o Discuss prograu priorities with U^Wo" 
panel for first half day of ^'L^ 
fgvletf peering. 



Discuss program priorities 
vith the staff. 



0 Co-Chalr review panel oeetlng 
vlth Dember of study section. 

o Durlog revlev Interject coaaaents 
about applicant's previous per- 
formances and substantive points. 

o Score of .proposal Is average of , 
scores assigncJ by each aieffibtr. | 



0 Vritc, Summaries <"plnk sheets") 
of review panel's action using 
sunsariea prepared by panel 
mcnters . 



I >Seetlng as a group, discuss each 
proposal in turn for about one 
half hour. Consider site visit 
report. If any. Make changes in 
proposed budget. level. If needed 
After discussing, approve or 
disapprove modified proposal by 
oajoiity vote of the psncl. tlien 
score proposal on a I to S scale 
by considering: 
- scientific cierit 
" feasibility 
-•competence of principal 

Anvestlgator. 



Present briefs at Council 
■iceting. Bring up any proposal 
vhlch: 

* received inadequate revlev. 

- vss disapproved by Bwre than 
- ' oD.e study secLlon rictnber, 

- la especially relevant to 
Branch's program. 



o Set pay Units, 
o Pay grants In priority score 
order. 



o Kcad study section 
sunmarles before 
Council ocrting to 
extent time Is 
•valloble. 

o riscuss the pro'^ 
posals one at sa 
tine for one half 
hour. Then ratify 
review panel score 
or Biodlfy j>roposal 
and rescore. 

o ^lembers bring up afy I 
other prupUK.lI they [ 
wish to discuss. 

o Over 952 of pro- 
posals ratified 
en bloc . 

o Council meeting 
chsfred by Insti* 
tute Dl rector. 



o Site visit every B to 9 months. 

o Kail each grantee a qut^stlonnalre 
three times during grant: with hla 
application form, six months after 
start, and at the end. Quest loniwire 
asks hln where his results ate useful, 
and how they will be Impleaented. 

o Branch staff rates conpleted projects 
on a four^polnt scale. Vsed as 
check on ttonltotlng. 
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Staffing Relationships 

Program Directors , Most program directors have an M.D. or a Ph.D. 
degree. They are selected by the Mental Health Services R&D Branch 
Chief, and approved by the Director, Division of Mental Health Services 
Programs. Each is assigned responsibility for a program area and works 
almost full-time in that capacity. The program areas are defined so 
that most overlap substantially \/ith at least one other program area. 
This way intrastaff dialogue is increased and greater flexibility in 
shifting manpower to priority problems is obtained. The program areas 
have been clustered into four groups, with each program director rotat- 
ing through a six-moiith terra as group coordinator. This also helps the 
program directors to learn other problem areas. The job of branch co- 
ordinator is filled jointly by the branch chief and the staff man respon- 
sible for Grants Review Administration. 

Review Panel Members . The branch chief recommends replacements for 
panel members from the community of researchers and practitioners when 
their four-year term expires. These nominations are then approved up 
the line to the Secretary of HEW. The branch chief serves as chairman 
of the review panel and chooses a co-chairman from the panel. 

Since the range of disciplines and problems covered by each panel 
is large, nominating panel members is a difficult job for the branch 
chief. In general, the best policy is to select two or three panelists 
from each discipline frequently relevant to the program, and at least 
two or more practitioners. It is especially important to have two or 
more extremely strong methodologists on the panel. 

National Advisory Mental Health Council Members . At least six are 
required by law to be authorities in mental health or science fields 
important to the Institute's concern. Two are ex officio representatives 
one required from DoD and the other from the Veterans' Administration — 
and as selected they tend to be scientists. Of the remaining six posi- 
tions, one or two are usually lay representatives and the rest scientific 
Advisory Council members are nominated by the Institute Director and ap- 
proved up the line to the Secretary of HEW. 
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IV, OFFICE OF ECONOMIC OPPORTUNITY 

OVERVIEW 

Like the Department of Agriculture and the National Institute of 
Mental Health, The Office of Economic Opportunity (OEO) supports both 
R&D and an action program in carrying out its mission, and like these 
agencies, the action part of the total program is larger than the 
research part. In the federal hierarchy, OEO is part of the Execu- 
tive Office of the President. In FY 1971, OEO spent $100 million 
on R&D, and $794 million on action programs, mostly on community action 
agencies and demonstrations. 

The budget for action programs is smaller now thar it used to be. 
This shift occurred in 1969, when the President announced a reorgani- 
zation of OEO. His policy was to have OEO concentrate on developing 
and testing new action programs, and move away from the operation and 
implementation of action programs. Once shown to be successful, new 
programs would be transferred to other federal agencies. In addition, 
OEO would assume a larger role in analyzing domestic policy issues 
pertaining to the poor. Thus, OEO was to have a dual role: staff 
to the Executive Office of the President on policy issues, and pro- 
ducer of verified, new programs. 

OEO's Office of Planning, Research and Evaluation (PR&E) plays 
a significant part in this strategy, since it is a principal R&D arm 
of OEO. In FY 1971 PR&E spent $26 million on R&D. Although other 
OEO units conduct R&D, their primary activity is arranging pilot 
programs and demonstrations, and not conducting R&D. The paradigm 
presented in this section will center on the PR&E part of OEO. 

The other major divisions of OEO are the Office of Operations 
($360 million). Office of Legal Services ($62 million). Office of 
llonlth .\f fairs ($100 million), and the Office of Program Development 
(v52 million). The Office of Operations runs OEO's Regional Offices 
and the Community Action Agencies funded by Title II of the OEO Act. 
It has a small decision-oriented research activity for dealing with 
service delivery problems. The Office of Legal Services has some 
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research funds ($5 million) , which are used for decision-oriented 
research, preparation of briefs for operating offices, and a univer- 
sity center. The Office of Health Affairs is primarily involved in 
deiDonstrations . The Office of Program Development (OPD) is functionally 
closest to PR&E. Its primary method of operation is to conduct 
"pattern" demonstrations and follow up with rigorous evaluations. The 
intention is to plan these demonstrations so that several together 
will contain a pattern of variations that together constitute an 
experiment. OPD tends to be more cntrepreneurially and less conceptu- 
ally oriented than PR&E carrying out its developiaental activity. 

. The Office of Planning, Research and Evaluation has pioneered 
in the use of carefully planned and evaluated social experiments 
a means of conducting R&D on social problems. While the idea of 
social experimentation is a simple application of empirical scienti- 
fic methods, its use in testing social programs is new and still under 
development . 

In accordance with tltls way of conducting R&D and its dual staff 
and developmental role, PR&E has been organized into three groups: 
Experimental Research, Policy Research, and Evaluation. Each of 
the groups has about twenty people. They spent, respectively, $16 
million, $5 million, and $5 million in FY 1971. Another reason for 
grouping the staff this way is that it puts people of like mind- 
set together. 

The Experimental Research group views its job as designing an 
interesting and important policy-relevant experiment and getting it 
going. Their operating style is much like OPD*s. 

The task in the Policy Research group is to do research into 
the underlying causes of poverty. Some '*basic" work is supported 
(in particular, a university center for poverty research), but in 
choosing problems and approaches the Policy Research group *s style 
is more policy-oriented than typically found in academic research. 
To a great degree, the staff is free to choose its own topics of 
research. The purpose of having this staff is to be able to produce 
!• ^rv^nt policy analyses on short notice. The PR&E Director believes 
that the probability of doing this kind of work before it is needed, 
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and having it end up right on the mark, is nearly zero. He believes 
that the only way to get good policy analyses done on a crash basis 
is to have an intramural staff that spends most of its time doing 
research. When an issue gets hot, someone on the staff will know 
what is known about the issue, and likely have given it some thou^jht. 
This person will then be able to produce a relevant policy analysis 
in a short time. 

Tlie task in the Evaluation group is to determine how well some- 
thing is working and what impact it is having. They are somewhat 
less concerned with why something works than is Policy Research, but 
not to the extent that reasons why a program is or is not working 
are avoided. On the contrary, Evaluation believes that diagnostic 
information is of much more use to decision makers than purely impact 
analysis . 

The Evaluation group is less concerned with getting something 
to work than is Experimental Research and more concerned with execut- 
ing complete designs that will hold up under counterexamination. 

Evaluation is concerned with the integrity, accuracy, and 
completeness of experiments and results. More than the other groups. 
Evaluation's mode of operation is to bring pieces of work to a conclu- 
sion, summarize the results, and dis-:ribute them widely. 

The essential features of PR&E's paradigpi, vhich will be called 
the Research and Evaluation paradigm, are the following: 

o Work is performed on a project basis. A wide range of 
performers are supported including private firms, 
nonprofit research institutes , practitioners , and 
academics . 

o Most project ideas are generated internally by intramural 

* staff and contracted out for performance, 
o Proposals are evaluated with a system of internal 
comnrittees. 

ot The organization plays a dual role: external as staff 
to the Executive Office of the President, and internal 
as developer and tester of new social action programs. 

o Management emphasis is placed on conceiving sound and 
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poHcy relevant experiments and evaluations and care- 
fully managing their performance. 

SUMMARY 

General Characteristics 

Primary output: Evaluated and tested solutions to 

selected social problems. 

Mechanism of support: Finite duration contracts are awarded 

to the winner(s) of an RFP competition. 
On some projects three semi-finalists 
are selected for Phase I contracts 
(design phase) , and one of these is 
selected for the Phase II (production 
phase) contracts . * 

Managerial emphasis: Most of the managerial effort is 

placed on conceiving and designing 
sound, policy-relevant evs luations 
and experiments. 

Staffing plan: Research and Evaluation unit of OEO 

consists of three staff groups: Evalu- 
ation, Exploratory Research, and Policy 
Research. Members of each group serve 
full-time in their positions, but 
there is a lot of staff interaction. 



Program ideas are generated by the 
Research and Evaluation staff. 



Program Planning 

Source of new ideas: 



"This feature is only being used on an experimental basis. 
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particularly its Director and the 
Policy Research group* The Director 
derives many of his ideas from ser- 
vice on presidential level policy 
task forces . 

Mechanism for planning: The Policy Research group does research 

and maintains links into the research 
community so that it will remain a 
source of fresh and sound ideas. 

Coordination: Plans for new programs are coordinated 

thro^'^h informal cortact at the program 
director's level, and jy the Director. 
The Director can be especially effective 
in coordinating programs because of 
his service on presidential level task 
forces . 
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Program Development 

Source of project ideas: 



Allocation of budget 
to projects : 



Evaluation of proposals; 



Within a program, project ideas are 
generated internally by the Research 
and Evaluation staff. Project ideas 
are then developed to an outline or 
"rough design" stage by a project 
management team of two or three staff 
professionals. The outline is written 
into an RFP by the project team. 

Budget decision is in effect set by 
an inhouse panel (Project Review 
Board) , which must authorize the 
writing of an RFP, and the Director 
of PR&E, who authorizes the creation 
of project teams. 

Bids received in response to the RFP 
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are reviewed and scored by an ad hoc 
technical review panel of mostly inhouse 
personnel. The winning contractor is 
selected by top-level OEO managers in 
consultation with project personnel. 

Especially during the design phase, the 
project management team works very closely 
with the contractor to make sure the 
desired product will be obtained. 

The project team writes a report on each 
contractor that is used in evaluating 
subsequent proposals . 



No formal procedures. 
ACTIVITIES 

In discussing the detail of program management activities, attention 
will sometimes focus on the Evaluation group, to the exclusion of similar 
functions done by the other two groups in PR&E. This simplification distorts 
the PR&E paradigm, but only slightly for three reasons. First, the pro- 
cedures used in Experimental Research are qualitatively similar to those 
used in Evaluation even though grants insteads of contracts are used. 
Second, since the primary responsibility of Policy Research is policy anal- 
ysis, discussion of how this is done would be irrelevant to the topic of 
managing practice-oriented R&D. Third, where the .operations of these three 
groups strongly interact, as in program planning, the role of all three 
will be discussed. 

At this point diS'Cussion will turn to the methods used for program 
planning in the Evaluation group of PR&E. 



Monitoring of 
performance: 



Evaluation of 
outcome : 



Program Evaluation 

Mechanism of 
evaluation: 
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Program Planning 

The Evaluation group conducts five kinds of evaluation projects: 
o Evaluations of experirnents conducted by the Exploratory 

Research group, 
o Impact evaluations of major national programs. 
o Comparative evaluations of different ways the same job 

is being done (natural experiments), 
o Analysis of data collected on other programs by other 

agencies . 

o Extremely short term analysis of questions asked by 

other agencies in the Executive Office of the President 
or OEO. 

There is no subject area restriction other than to the needs of the low- 
income population. Programs in housing, manpower, education, health, or 
welfare may be evaluated. 

The Evaluation Group has no explicit rationale for generating new 
evaluation program and project ideas and choosing which to develop, except 
that at a minimum every one of PR&E's experiments is evaluated. There is 
no attempt to develop a long-range plan. Nevertheless, all program and 
project ideas that are eventually developed originate in the Evaluation 
group or in another part of PR&E. Therefore, the long-run effectiveness 
of the Evaluation group depends strongly on the quality of the ideas that 
the whole PR&E staff generates and its ability to choose wisely among 
them in selecting tbe few that are developed into projects. In the long 
run, the quality of the ideas generated and of the choices made among them 
depends upon: 

o The background and quality of the staff. 

o The steps that management takes to encourage staff inter- 
action, and • 

o The contacts that staff have with the external environment* 

External Contacts . One primary set of contacts is provided by OEO's 
position in the Executive Office of the President (EOP) . Because of this, 
position, the Director of Research and Evaluation is called to serve on 
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policy task forces and cousult with personnel in the White House, Office 
of Management and Budget, and other EOP organizations. These contacts 
affect the Director's priorities among problems and are a source of new 
ideas that he translates into new programs. 

Another external contact is provided by a managerial policy to main- 
tain sufficient intramural capacity within the Evaluation group that data 
collected on programs sponsored by other agencies can be analyzed intern- 
ally. Often these analyses produce new results and sometimes lead to 
ideas for new programs. 

A third set of contacts is provided indirectly through the Policy 
Research group. As mentioned previously, one of the Pol:' ry Research staff 
activities is to do research into the underlying causes of poverty. Since 
the knowledge base is weak in this area, the staff works in an exploratory 
mode much of the time by thinlcing through novel ideas, and by keeping in 
contact with what is going on in their specialty. One means of contact is 
traveling frequently throughout the social research world. Another is to 
arr^inge research grants wi^- extramural researchers, and use them as con- 
sultants and tapf'> ;u;tc uie research community. As a product of these con- 
tacts and tb*-ir own research, the Policy Research staff generate ideas for 
r:iw i//Cgtams. Since the whole Research and Evaluation operation is small 
(less than 80 people) , the Policy Research staff can interact frequently 
with the Director and with members of the Evaluation staff, and conmiunicat 
these ideas directly to the people who can implement them. 

The Policy Research staff is an especially good source of program 
ideas, because their overriding objective is doing sound policy analysis 
on short notice. To produce sound analysis, the Policy Research group 
must know and develop knowledge about which are the most sensitive and 
crucial system variables. Thus, their ideas are tempered by knowledge of 
what is likely to have the most effect. 

To assure that this idea generation and research function of the 
Policy Research group is not lost to pressures for more policy analyses, 
the PR&E Director carefully regulates the number of requests for analyses 
given to Policy Research staff. The overflow of requests ' 



OEO has authority to award both contracts and grants. 
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are given to the Policy Analysis staff, a small group of generalists who 
report to the Director of PR&E and are capable of doing analyses on short 
notice. This staff typically numbers four or five professionals with 
backgrounds in a variety of fields such as law, economics, or history. 
This staff is distinguished by its ability to do analysis on a wide range 
of topics on short notice. The PR&E Director believes that the availability 
of such a staff, which acts as a buffer for the Policy Research group, is 
essential to PR&E's long-run ability to generate new ideas and maintain its 
research base* 

Staff Interaction . The idea generation and filtering process is also 
enhanced by the deliberate, managerial policy of maintaining differences 
between the perspectives taken by the three Research and Evaluation groups, 
and on frequent occasions, bringing them together to work on common problems. 
The perspective in Policy Research is knowledge-oriented and decision- 
oriented. They are concerned with what is known and what needs to be known. 
The perspective in Experimental Research is to get some experiments going, 
and produce results even if it means some sacrifice in methodology. The 
perspective in Evaluation is to press for changes that would improve the 
quantity and integrity of measured experimental results. During the formative 
stages of new programs, the Director of PR&E brings together staff from each 
of these groups. The complementary nature of the dialogue produced serves 
to keep all parties open to new alternatives, and helps each recognize over- 
sights. The result is better quality program ideas and formulations. The 
Director makes sure that this process operates by resolving issues at his 
level. He forces interaction among all the parties involved, and does not 
allow insular agreements to build up at the working levels. 

Since the rest of OEO is organized by problem area and PR&E is organized 
by research function, there is also a difference of view between PR&E staff 
and the rest of OEO. This differential minimizes the chances that PR&E becomes 
tied in thought to existing operating programs. 



The Policy Analysis staff is just being instituted in PR&E. 
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Staff Quality , The third means of enhancing the idea generation and 
selection process is to maintain quality and diversity in the three PR&E 
staff groups. 

The Evaluation staff consists half of historians and sociologists and 
half of statisticians, systems analysts, and operations researchers. A 
few of the staff specialize in one type of evaluation or subject area, but 
most do not. A few specialize in doing the short-term analysis of questions 
posed by other agencies, because this type of work takes special skills. 
The Policy Research staff are mostly economists, but some are sociologists, 
psychologists, political scientists, and computer scientists. They tend to 
specialize in subject areas, for example, housing. The Experimental Research 
staff also have academic backgrounds, but are less research-oriented and more 
engineering-oriented in their desire to construct and investigate in an 
empirical way. Most have a social science background • 

While the pnrase, program idea, has been used to this point in discus- 
sing the planning process, in actuality programs do not exist in PR&E as 
managerial entities. Programs could be defined by sorting projects in the 
three PR&E groups into categories of related projects, but there is no mana-- 
gerial recognition of programs as such in the assignment of managerial re- 
sponsibility or budgeting. Rather, idea formulation occurs at the project 
level, where a project is a contractable unit of work to be managed by one 
of the three PR&E groups. Projects may be strongly related, as is the case 
when an experiment and an evaluation are coupled, but such a strong coupling 
does not always exist. More usually project ideas are generated sequen- 
tially in the course of elaborating a central theme. Thus, an effort to 
understand the role of incentives in the delivery of health care might in- 
clude projects on income maintenance, various forms of insurance, physicians* 
fee adjustments, and others done over a period of time. Successive pro- 
jects would be devised by the PR&E staff to articulate the variations and 
aspects of the central theme with the purpose of understanding it more 
clearly. By the nature of social experimentation and evaluation many of 
these projects take several years to complete and are very large in size, 
sometimes up to several millions of dollars. At any one time only a few 
will be supported. 

Because of this large size and the form of program organization, it is 
;tri.y difficult to sort management activities into the categories. Program 
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Development, which involves elaborating current themes, and Program Planning, 
which involves generating new themes. In PR&E, the processes used to decide 
what the next round of projects will be in Program Development are indistin- 
guishable from the way new themes are generated. 

This ambiguity will be avoided by reassigning the project idea gener- 
ation stage of Program Development to Program Planning. Thus, Program 
Development will include only the activities of taking a project idea, 
developing it into a set of tasks, perforaing the work, and reviewing the 
outcome. 

Program Development 

For the first three kinds of evaluation projects that the Evaluation 
group does — experiment evaluations, program evaluations, and comparative 
evaluations — the Program Development process consists of four sequential 
phases: 

o A preliminary phase where OEO decides what the evaluation 

design should be; 
o Phase J, the design phase, where the contractor finalizes the 

evaluation design; 
o Phase II y the production phase, where the evaluation is 

carried out; and, 
o A follow-vqp phase y where additional analyses of data and 

dissemination occur. 

The last two kinds of evaluations done — data analyses and short-term 
policy analyses — are performed intramurally by the Evaluation group staff. 
Usually, these go through the same four phases, but the management control 
process is much less formal than for the first three kinds of evaluations 
since the latter are all performed extramurally . -In what follows, discus- 
sion will be limited to managing the first three kinds of evaluations. 

In general, PR&E lets one contract for both Phases I and II of an 
evaluation project. At the present time, however, PR&E is experimenting with 
the procedure of awarding several Phase I contracts, and a separate contract 
for Phase II to the best of the Phase I contractors. The two-phase procedure 
.is intended to provide additional incentive for the contractors to cooperate 
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with, the project team during the evaluation design phase. A high level of 
cooperation is felt to be exceedingly important for best results. The 
procedure also gives additional information about the capability of the 
contractors to perform before the final commitment of the bulk of project 
funds must be made. In describing Program Development this Phase I/Phase II 
procedure will be included as an option that PR&E sometimes uses even though 
it is only in an experimental stage of development because it illustrates 
the importance PR&E places on getting contractors to work with the project 
team. 

The Preliminary Phase of project development is carried out intramurally 
by a team of at least two people. The same two people generally follow a 
project through the first three phases. At least one of these people is an 
analyst, and at least one is a subject-knowledgeable person. The team some- 
times uses consultants for short periods to cover gap areas. Sometimes 
exploratory surveys are done. The" Preliminary Phase lasts from two to four 
months depending on the complexity of the project, and other factors. The 
preliminary phase must begin well before the experiment or program to be 
evaluated begins if preintervention data are to be collected. Since the 
Phase I contract (on which the evaluation design is finalized) lasts between 
a few months to a year, the Preliminary Phase must begin more than half a 
year earlier, or roughly a year or more before the intervention starts. For 
example, on OEO's Voucher Experiment scheduled to begin in September 1972, 
the Phase I evaluation contract will probably be let by October 1971, so 
that Phase I can be finished by January 1972, and a contractior selected for 
Phase II by February 1972. Preintervention data will be collected in April 
1972 by the Phase II contractor. To fit this time schedule, the Preliminary 
Phase had to start in April 1971, a year and three months before the experi- 
ment. 

The purpose of the preliminary phase is to answer four questions: 
o What is to be found out? 
o How can it be found out? 
o Who will use the results? 
o About how much will it cost? 
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Because the reason for doing an evaluation is to develop information for 
decisionmakers, and not to do research, the team spends most of its time 
on two of these questions: Who will use the results, and how can these 
be found out? On the question of "what is to be found out," the team tries 
hard to separate the "nice" kind of things to know from the essential things 
to know — the information decisionmakers can use. The "how" question in- 
cludes questions such as: Is a national sample necessary, or will a selected 
site be enough? Must evaluations be done before, during, and after the 
intervention to get usable results, or is a post-experiment evaluation suffi- 
cient? Will a pattern of demonstrations suffice, or is a controlled inter- 
vention required? 

The project team determines answers to these questions and incorporates 
them in an RFP for the evaluation project. The project team works hard in 
perfecting the RFP, for sometimes as many as five or more versions are written 
before a final product is obtained. Successive versions are critiqued for 
technical substance and terras of contract by the Director of Evaluation, or 
the Director of PR&E, or both, depending on their interest in the problems. 
The final product specifies the evaluation design in outline form, the 
criteria to be used in evaluating proposals, and the weight in numerical 
score that each criteria will have in selecting the winning contractor (s) . 

PR&E's process for reviewing RFPs and proposals begins with a management- 
oriented review that occurs before RFP-writing begins in earnest. This review 
is conducted by the Project Review Board , a standing committee consisting of 
the PR&E Director, ten PR&E staff members selected by him, one representative 
from the General Counsel* s Office, and one from the Procurement Office. The 
purpose of this review is to check that the necessary funds have been ' id- 
geted and that the estimated cost is appropriate for the proposed project. 
The Board also determines if a contract is the appropriate instrument and 
what the terms of contract should be (cost plus fixed fee, or fixed price, 
or some other terms). 

If the estimated project cost is over $300K, an ad hoc Procurement 
E valuation Board is convened near the end of the preliminary phase to review 
the final drafts of the RFP. The Board is responsible for reviewing the 
criteria to be used in selecting the winning contractor (s) , and developing 
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an instrument for evaluating proposals with respect to the criteria. The 
instrument is a detailed list of questions that proposal evaluations will 
use in assigning scores for each criterion. The purpose of the Board's 
review is to increase the assurance that the winning contractor (s) will be 
selected impartially. The Board members are selected by the Deputy Director 
of OEO. Included are the Director of Procurement, the project team leader, 
and an agency-wide cross section of seven to ten senior-level staff people. 

If the estimated project cost is less than $300K, the process for re- 
viewing the criteria to be used in evaluating bidders is less formal. 
Instead of convening a Procurement Evaluation Board to accomplish this re- 
view, it is done by PR&E top management at the same time they are reviewing 
the final versions al the KFP for substance. 

Whether the erjtimated cost is over or under $300K, the final version 
is reviewed for its technical substance and legal conformity by OEO managers, 
specifically the Director of PR&E, the OEO General Counsel, and the Director 
of the Procurement Office. If the RFP is acceptable to all of these reviewers, 
the Director of PR&E releases it for inclusion in Commerce Business Daily. 
In addition to this means of distribution, the RFP is also sent directly to 
individuals or firms that the PR&E management or staff think would be excep- 
tionally able performers. 

Proposals received in response to the RFP are evaluated with a committee 
review system. If the estimated project cost is over $300K, the Procurement 
Evaluation Board, the same panel that has reviewed the RFP, does the first 
level review. If the estimated project cost is less than $300K, a Procure- 
ment Evaluation Board is convened to perform the first level review. The 
panel inembers are chosen by the project team leader with the concurrence of 
the Procurement Office. Included are the contracting officer for the project, 
senior- level staff people from throughout the agency, and sometimes a few 
from outside OEO, either government or non-government. 

The Procurement Evaluation Board members rank the proposals on tech- 
nical merit according to the criteria set in the RFP. The exact procedures 
used for assigning scores are not standard but as a general rule each RFP 
is read and scored by four Board members. The proposal's technical merit 
score is the average of its reader's scores. On over-$300K projects, the 
Board members are guided in assigning scores by the list of questions developed 
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during the RFP writing stage. On under $300K projects, the Board operates 
less formally by giving greater discretion to the Board members in assigning 
scores. Board members are not left entirely to their own, however, because 
the Board does reach some agreement on guidelines that members will use in 
assigning scores. 

After technical scores have been assigned the Board sees the cost 
portion of proposals for the first time. The Board then determines which 
set of projects are in the "competitive range." The competitive range is 
a vaguely defined concept, but roughly consists of those proposals which 
are high in technical merit but not too high in cost. The Board reaches 
agreement on what range of technical scores and costs constitute the compe- 
titive range through consensus agreement. On the average about five bid- 
ders are included but sometimes ten 6r more are included, especially on 
expensive procurements. The technical rating of particular proposals may 
be reassessed during this time. 

The bidders in the competitive range are then invited in to discuss the 
strengths and weaknesses of their proposals with the project team and the 
Director of Evaluation, and with the Procurement Office. During these 
meetings the project team and the Director of Evaluation are able not only 
to exchange information of technical substance but to assess the sincerity 
and commitment of each bidder. The Procurement Office gives bidders until 
a predetermined date to revise either or both their cost or technical 
proposals. 

Revised cost and technical proposals received from these bidders are 
then rescored and re-evaluated by the Procurement Evaluation Board as its 
final determination of rank order according to technical merit. For this 
evaluation each proposal is read and scored by each Board member. 

If the estimated project cost is under $300K, the final decision on 
which contractor (or contractors if the Phase I /Phase II device is being 
used) is made by the Director of PR&E but he relies to a great extent on the 
opinions and judgments of the project team leader and the contracting 
officer for the project. In general, contracting officers tend to favor the 
lower cost proposals in the competitive range, and project team leaders the 
technically most meritorious in the competitive range creating a tension 
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between them that is resolved by the Director of PR&E. In making his 
choice the Director of PR&E is also influenced by the Director of Evalua- 
tion. 

If the estimated project cost is over $300K, the final decision on 
winners is made by the Source . Selection Board. This Board consists of the 
Deputy Director of OEO, the Director of PR&E, and the General Counsel. 
The Board hears a presentation by the project team leader summarizing the 
results of the Procurement Evaluation Board deliberations. Both the 
Procurement Evaluation Board's findings in determining which bidders were 
in the competitive range and their findings on revised proposals from 
bidders in the competitive range are included. After discussion the Source 
Selection Board reaches a consensus decision on the winning contractor (s) . 
If the Phase l/Phase II procedure is being used, the Board decides how 
many Phase I contracts will be awarded^ 

Evaluation contracts are very closely monitored during their performance. 
A $1 million evaluation will be managed full-time by a two- or three-man 
team, especially during the design phase. Usually, it is the same team 
that wrote the RFP. The project team works closely with the contractors 
iri developing questionnaires, and site-visits both the contractor and the 
data sites. The project team is very active and aggressive in managing 
evaluation contractors so that the final product will satisfy OEO*s needs. 
PR&E believes that close monitoring is essential to obtaining results that 
decision makers can use. 

If during the course of an evaluation project a major change in con- 
tract rules or scope is desirable, this need is reviewed by the Project 
Review Board and then negotiated with the contractor. 

If the Phase l/Phase II procedure is being used, each of the Phase I 
contractors submits a proposal for Phase II at the end of Phase I. The 
Phase II proposals are evaluated using the same committees and system of 
review used for Phase I. 

At the end 'of an evaluation project the contractor writes a final re- 
port that is released by PR&E for public consumption. The contractor also 
submits all the data collected on the project. These data are made avail- 
able for researchers who have use for it. 

ERIC 
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Gompleted projects are disseminated in two ways. The contractor 
submits a final report on his findings that is then released. The project 
team also writes a report that summarizes the policy relevant findings 
and indicates where additional study is needed to get additional policy 
relevant results. 

Sometimes a grant or contract is awarded for this additional study. 
Either the evaluation project contractor, if he performed well, or another 
contractor may be selected for this work. Otherwise, the additional study 
is done by the Evaluation staff. The need to do additional studies is not 
indicative of a poorly-designed evaluation, or poor performance. Some- 
times policy questions arise during the course of performance that were not 
foreseen at the beginning. 

OEO's evaluation results are strongly coupled into policymaking be- 
cause of its position in the Executive Office of the President. Since the 
Director of Research and Evaluation serves on task forces and consults 
with White House and 0MB people on a regular basis, he carries evaluation 
results directly to decisionmakers. This is an important feature of the 
OEO Research and Evaluation paradigm. 

Within sixty days after the close of an evaluation contract, the pro- 
ject team evaluates the contractor's performance and submits a report to 
CEO's procurement division. These reports are stored, and recalled later 
for use by review panels when the contractor applies for another contract. 

A diagram of the program development process appears in Figure 11. 

ORGANIZATION 

Organizational Relationships 

CEO's Office of Planning Research and Evaluation is formally composed 
of three units: Experimental Research, Policy Research j and Evaluation. 
These groups are actually only loose confederations of specialists doing a 
similar job, and are not highly compartmentalized. There is a great deal 
of interplay between the staffs. A chart of OEO appears in Figure 12.. 



I 



■98- 



S 



9 




O 1 
• Q « o )« S 




• > S U 4 M 




w u $t m CO 






• « • U t4 

^ • H 4 




• 4 4i 4J t) 0 

■«» • « <J d fc- « 




•J M O «| (1 9 

s s**^ ■> y 

^ S 0 o »» 0 0 

Ck£ Ck,(W 4J 


• 

B 
0 
«H 


O 0 





O M 
JB 4 « 

"•S 3 " 

13 U 

*J W M C II 

4 *• P tM u 

^4 -H . fci 



"I V O U 
4 ^ CV 3 (0 



g. > M 
O ^ V 41 

O. CJ «J u 



42* 



V i< q U 

or O *i 

V u ti >s 
u a, > 

p. «3 iH 
U 4 « 



V 4J »i ^ u 
B cn X B CO 

V O «i jB O 
Vi ti -r* U U 

S3 U < 

O <i) U I 

^4 u c u u I 

V «j m o u I 
«t (0 M n I 

> V o. U I 

es I I 



U V 
U U 
C -H 



O «4 u 



o o: 



J= O V V 



(I O B h 



> X o 
i 4 «] 



O 4J 
B ^ 



(« • 

B ^ 

^ . ifl u O ^ ft] 
O. 09 »j ^ tj n ■ 

" ~ U U 



• »4 



Q 



B U O U -H 
a* C 4 U 



B « 



B U t( OJ « 



4j «j «) o ; 

'J1 01 > o. ^ 

O o. iq O 4 

. 0 41 a « 

^ u e 

V V i 

u ji c ^ i 

o «i O u ( 

(J iq 

n u • a 

• B CI a ^ 

U «M B OJ 

OJ u ca X f 

4J ^9 Wl 4J > 



** c u 

0 o 

(0 u 

n B u 

4/0 4 

»-1 ^ M 

01 >4 C 

M u O 

4 V U 



I -.^ i< 
I ^ o c 
o c 



o: 5 V 
o u u 

• 41 M 

C3 T3 
O tl (O W 
•I 4 
O 

U3 ^ JO n 

« 41 U B 

■a x: o 



4) O «M C>> w4 . _ 



IX, 

bu O 



y (2 



4J a 

B 
O 



B 

U £ 



41 C3 U >1 • 

•H 01 U r-l-'T3 U iH 

> .H in *4| B 4 .H 

u CJ c o: o £ ai 

o: a: M Oi (/» 4i 3 



15 



U 00 



4 « 

C ^3 *J 

O B ^ 

O O «) 

n V 

3 4 v4 e- 

V ^ V Li 

^ « fl 

V O B 9 O 
t\ U U 

V «M —* 

.B ffl iJ 



« 5 a: 5 
•5 ♦ I I 



^ « t> > 

^ 94 fH (U 

kt i-H 

iJ t> Q W 



s 

tn 
«> 

U 
« 



4 S 

as 

ffl t> 

V c 
4 41 4 

j: u I 
u 4 i 

' 

•Sill 



B 4t ^ <W O 

•H «n > u 

> O 4 60 

j: «i u c 

U 4J £ U *4 

t> B >% ^ U 

fH O r-( 0 4 

4t M OJ 14 ki 

M eg o. u 



> C £ 

M O iJ 

*J -H 00 

^ ffl c 

U B) 4> 

4> 3 kt 

C «) «6 
O v4 
U T3 - 
4 

B W -H 

^ o n 

W4 ffl 

« o 

0 o. • 

&k G o n 

O kt 0 



^ O ki 0 ki 



ki C I 

fc4 o o 

4 M ki 

O. «> Oi 
n 

B 9 £ 

O U 4J 

^ n ^ 

«J » . 

3 *J 

i-H 4J 4J 

d n 01 

> o C 



2 > 



o « cn 

u U 

U ^ CJ 

4) Q 

ki £X T3 



» O k> 

t fij ffV • 

iJ OA o 

B « o »r 



ki U M O 



r fir u M u « > o« 



•« 4 



3 • B , 

C 'O • 01 I 
^ Wl ^ w< ' 

43 ki U L 

>» o »3 e» 

ea o 13 I 

00 a I 
c • i-l 

o c 4 J 

k. a o B» 



4 ki CO ki I 

n (J I 
Q i> Wl (U , 

« U 9 «^ ' 
U B O 
H 4> <0 Ai 1 



I 




■H B 4 41 W« 

V -rt B 4* U 

0 i:i o ki 4 
■H n • u ^ ki 

u B B u 

1 CO U ffl ft . 

•H «M ^ ^ 4> O >s 

m m u u Wl 

3 c u *4 *4 n 

•m ^ > n ft, ai 

U (A 

M m c V m c V 

U U O ki 4J o u 

Q O ^ rt w 

> k» «4 •ij U B 



44 V 

. > 4 



«i u n 

O «J B 



U kl iH 

- 00 

•C ■ «J 

«J 4 U 

« 4 . B 

■ 91 4 Wl M 



4 4> B ki 01 



U O P M -*4 



W b M Wl 

^ V B 

4 M ' 
D.«-l > 

J4 8 4 

tJ - " 

»K M I/) 



COD. 
4 U 0 
B U 



U V 
V O U 

M Wl 
Wl 4> 9 
U CW«M 



U ffl ffl 4> f-4 



ERIC 



41 



^ u b! 



4 ffl ffl i 

> 41 4 V I 

4 X •! 

0 ti 



I «M 5 



M « • B 



• M H O 4 fl 

w 4 u ji 



1^ 4 b « O M 



w 4) 



kl W 4> 



fl 
4> O 



4 -ffl 



8:5 

►H — \ w >-< n 
« p fc> n. M 
> ^ 9 H 4 

UI O. O M W 



-99- 







M 


CO 


O 


M 


4J 


•H 


O 




Q) 




xr 










CJ U 


4J 




CO 




CO 




<3 


O 33 



o 

O 

4J 
M 
CO 

0 
O 
•H 
4J 

cd 

N 













O O 




4J M 








Q) a 








•H O 0 








Q) O 




• U rH 




*J CU 




CO U-< > 




CO Q) 




<J O Q 



bO 

O 



M 

S, 

•H 

P4 





M 


o 


O 




+J 


o 


O 




Q) 




V4 




•H 


o 


Q 


4J 




o 


>^ 


Q) 


4J 


M 




•H 




Q 


Q) 








ERIC 









M 






O 


Q) 




4J 






O 


o 




Q) 






M 






•H 


o 










• 


a 


CO 






a 


CO 




o 


CO 




•H 


<J 


o 


4J 










o 




•H 




4J 








60 




a rH 




•H td 




S > 


M 




o 








o 




Q) 




M 




•H 


O X! 




O 




0) M 




o (d 




•H 0) 


CO 


CO 


CO 




< 


O P4 













td 




4J 




W 


a 




td 


4J 




a 






a 




•H 






00 




td 




a 








1 










O 




M 


O 


td 


•H 




i-l 


• CO 


O 













td 




4J 




a 






•g 


•H 




M 


M 


td 


0) 


0) 




CO 











-100- 



A Director has been chosen for the Exploratory Research group and for 
the Evaluation group, but the Director of Research and Evalufc;:.ion leads 
the Policy Research group. 

The Director of PR&E is also served by two other small groups: the 
Policy Analysis staff and the Planning and Management staff. The Policy 
Analysis staff is responsible for handling requests for policy analysis on 
short notice when the needed competence is missing in the three main func-. 
tional groups or their workload is too great. The Planning and Management 
staff performs the administrative, budget, and operational planning tasks 
that PR&E needs to have done. 

Project Review Board . The Director of PR&E chairs this Board and chooses 
approximately ten (the number varies over time) PR&E senior staff as members. 
Generally J these senior staff are in the GS-15 salary range. In addition 
a representative from Procurement Office and the General Counsel's office, 
chosen by these offices, also serve. This is a standing Board. 

Procurement Evaluation Board . The Deputy Director of OEO chooses between 
seven and ten senior- level staff to serve on this Board. One of these mem- 
bers is the evaluation project team leader r In addition, someone from the 
General Counsel's office and the Procurement Office also serve. -.Anew 
board is chosen for each procurement reviewed. 

Source Selection Board . The Source Selection Board consists of the 
Deputy Director of OEO, the Director of PR&E, and the General Counsel of 
OEO. 



This staffing function is not yet fully -^.inplemented in PR&E. 



